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DESCRIPTION 

Signal Processing Method and Apparatus and Recording Medium 



Technical Field 

This invention relates to a signal processing method and apparatus and to a 
recording medium. More particularly, it relates to a signal processing method and 
apparatus and to a recording medium which takes the difference between the signals 
as detected by the sensor and the real world into account. 

Background Art 

Such a technique is widely exploited which detects events in the real world by 
sensors and which processes sampling data output by the sensors, such as data 
associated with pictures, speech, temperature, pressure, acceleration or odor. 

For example, a picture obtained on imaging an object moving in front of a 
predetennined still background by a video camera is subjected to motion blurring in 
case the object is moved at a higher velocity. 

For example, a picture obtained on imaging an object moving in front of a 
predetermined still background by a video camera employing a CCD is subjected to 
motion blurring in case the object is moved at a higher velocity. That is, when the real 
world is detected by a CCD as a sensor, the picture, as sampling data, undergoes 
distortion. 
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The conventional practice in suppressing this motion blurring is to increase the 
speed of e.g., an electronic shutter to provide for shorter light exposure time. 

However, in raising the shutter speed in this manner, it is necessary to adjust the 
shutter speed of the video camera before proceeding to photographing. So, the blurred 
picture, previously acquired, cannot be corrected to obtain a clear picture. 

On the other hand, if an object is moved in front of a stationary background, not 
only motion blurring due to mixing of no other than the picture of the moving object, 
but also the mixing of the background picture and the moving object occurs. In the 
conventional system, no consideration is given to detecting the mixing state of the 
background picture and the moving object. 

Moreover, the infonnation of the real world having the space and the time axis 
is acquired by a sensor and made into data. The data acquired by the sensor is the 
infonnation obtained on projecting the information of the real world in the time and 
space of a lower dimension than the real world. So, the information obtained on 
projection is distorted due to projection. Stated differently, the data output by the 
sensor is distorted relative to the infonnation of the real world. Moreover, the data, 
distorted by projection, also includes the significant information for correcting the 
distortion. 

In the conventional signal processing on the sampling data, acquired by the 
sensor, the sampling data obtained by the sensor is deemed to be the most reliable 
data, such that, in subsequent data processing, such as transmission, recording or 
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reproduction, it has been a sole concern to realize the state of data which is as close 
to that of the original data as possible, in consideration of deterioration caused by e.g., 
data transmission. 

Heretofore, the sampling data output by the sensor is deemed to be the most 
reliable data, such that no attempt has been made to prepare data higher in quality than 
the sampling data or to perform signal processing of extracting the significant 
inforaiation buried by projection. 

Disclosure of the Invention 

It is therefore an obj ect of the present invention to provide for adjustment of the 
amount of motion blurring contained in detection signals of a blurred picture. 

It is another object of the present invention to enable detection of a mixing ratio 
indicating the state of mixing of plural objects such as a background picture and a 
picture of a moving object. 

It is yet another object of the present invention to provide a signal processing 
apparatus in which sampling data output by a sensor may be freed of distortion or the 
significant infonnation can be extracted from the sampling data, for example, to 
provide for adjustment of the amount of motion blurring contained in the detection 
signal if the sampling data is that of picture. 

The present invention provides a picture processing apparatus for processing 
picture data made up of a predetennined number of pixel data acquired by an imaging 



device having a predetenmned number of pixels each having an integrating effect, the 
picture processing apparatus including processing unit decision means for deciding, 
based on area infonnation specifying a foreground area made up only of foreground 
object components making up a foregroimd object in the picture data, a background 
area made up only of background object components making up a background object 
in the picture data, and a mixed area which is a mixture of the foreground object 
components and the background object components in the picture data, the mixed area 
including a covered background area foraied at a leading end in a movement direction 
of the foreground object, and an uncovered background area fornied at a trailing end 
of the foreground object, a processing unit made up of pixel data lying on at least a 
straight line extending in a direction coincident with the direction of movement of the 
foreground object from an outer end of the covered background area to an outer end 
of the uncovered background area, centered about the foreground area, normal 
equation generating means for generating a non-nal equation by setting pixel values of 
pixels in the processing unit decided based on the processing unit and a dividing value 
which is an unknown dividing value obtained on dividing the foreground object 
components in the mixed area with a predetermined dividing number, and calculating 
means for solving the nonnal equation by the least square method to generate 
foreground object components adjusted for the quantity of movement blurring. 

The present invention also provides a picture processing method for processing 
picture data made up of a predetermined number of pixel data acquired by an imaging 
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device having a predetennined number of pixels each having an integrating effect, the 
picture processing method including a processing unit decision step of deciding, based 
on area infonnation specifying a foreground area made up only of foreground object 
components making up a foreground object in the picture data, a background area 
made up only of background object components making up a background object in the 
picture data, and a mixed area which is a mixture of the foreground object components 
and the background object components in the picture data, the mixed area including 
a covered background area foraied at a leading end in a movement direction of the 
foreground object, and an uncovered background area formed at a trailing end of the 
foreground object, a processing unit made up of pixel data lying on at least a straight 
line extending in a direction coincident with the direction of movement of the 
foreground object from an outer end of the covered background area to an outer end 
of the uncovered background area, centered about the foreground area, a nonnal 
equation generating step of generating a nonnal equation by setting pixel values of 
pixels in the processing unit decided based on the processing unit and a dividing value 
which is an unknown dividing value obtained on dividing the foreground object 
components in the mixed area with a predetennined dividing number, and a calculating 
step of solving the normal equation by the least square method to generate foreground 
object components adjusted for the quantity of movement blurring. 

The present invention also provides a picture processing program for processing 
picture data made up of a predetennuied number of pixel data acquired by an unaging 
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device having a predetennined number of pixels each having an integrating effect, the 
picture processing program including a processing unit decision step of deciding, 
based on area infonnation specifying a foreground area made up only of foreground 
object components making up a foregroimd object in the picture data, a backgroimd 
area made up only of background object components making up a background object 
in the picture data, and a mixed area which is a mixture of the foreground object 
components and the background object components in the picture data, the mixed area 
including a covered background area fornied at a leading end in a movement direction 
of the foreground object, and an uncovered background area fonned at a trailing end 
of the foreground object, a processing unit made up of pixel data lying on at least a 
straight line extending in a direction coincident with the direction of movement of the 
foreground object from an outer end of the covered backgroimd area to an outer end 
of the uncovered background area, centered about the foregromd area, a normal 
equation generating step of generating a nonnal equation by setting pixel values of 
pixels in the processing unit decided based on the processing unit and a dividing value 
which is an unknown dividing value obtained on dividing the foreground object 
components in the inixed area with a predetennined dividing number and a calculating 
step of solving the nornial equation by the least square method to generate foreground 
object components adjusted for the quantity of movement blurring. 

The present invention also provides a signal processing apparatus for processing 
detection data, acquired every predetemiined time period by a sensor made up of a 
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predetennined number of detection elements having time-integrating effects, every 
predeteraiined time period, the signal processing apparatus including foreground 
sample data extracting means for extracting the sample data present in detection data 
before or after considered detection data where there exists considered sample data 
which is the sample data under consideration, as foreground sample data 
corresponding to an object proving the foreground in the real world, background 
sample data extracting means for extracting the sample data present in detection data 
lying after or before the considered detection data where there exists considered 
sample data which is the sample data under consideration, as background sample data 
corresponding to an object proving the background in the real world and detection 
means for detecting a mixing ratio of the considered sample data based on the 
considered sample data, the foreground sample data and the background sample data. 

The present invention also provides a signal processing method for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
predetennined number of detection elements having time-integrating effects, every 
predetermined time period, the signal processing method including a foreground 
sample data extracting step of extracting the sample data present in detection data 
before or after considered detection data where there exists considered sample data 
which is the sample data under consideration, as foreground sample data 
corresponding to an object proving the foreground in the real world, a background 
sample data extracting step of extracting the sample data present in detection data 
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lying after or before the considered detection data where there exists considered 
sample data which is the sample data under consideration, as background sample data 
corresponding to an object proving the background in the real world and a detection 
step of detecting a mixing ratio of the considered sample data based on the considered 
sample data, the foreground sample data and the background sample data. 

The present invention also provides a signal processing program for processing 
detection data, acquired every predetennined time period by a sensor made up of a 
predetennined number of detection elements having time-integrating effects, every 
predetermined time period, the signal processing program including a foreground 
sample data extracting step of extracting the sample data present in detection data 
before or after considered detection data where there exists considered sample data 
which is the sample data under consideration, as foreground sample data 
corresponding to an object proving the foreground in the real world, a background 
sample data extracting step of extracting the sample data present in detection data 
lying after or before the considered detection data where there exists considered 
sample data which is the sample data under consideration, as background sample data 
corresponding to an object proving the background in the real world and a detection 
step of detecting a mixing ratio of the considered sample data based on the considered 
sample data, the foreground sample data and the background sample data. 

The present invention also provides a signal processing apparatus for processing 
detection data, acquired every predetermined time period by a sensor made up of a 



predetennined number of detection elements having time-integrating effects, every 
predetennined time period, the signal processing apparatus including still/movement 
decision means for deciding still/movement based on the detection data, and detection 
means for detecting a mixed area containing sample data having plural real world 
objects mixed together based on the results of discrimination. 

The present invention also provides a signal processing method for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
predetennined number of detection elements having time-integrating effects, every 
predetennined time period, the signal processing method including a still/movement 
decision step of deciding still/movement based on the detection data, and a detection 
step of detecting a mixed area containing sample data having plural real world objects 
mixed together based on the results of discrimination. 

The present invention also provides a signal processing program for processing 
detection data, acquired every predetermined time period by a sensor made up of a 
predetennined number of detection elements having time-integrating effects, every 
predetennined time period, the signal processing program including a still/movement 
decision step of deciding still/movement based on the detection data, and a detection 
step of detecting a mixed area containing sample data having plural real world objects 
mixed together based on the results of discrimination. 

The present invention also provides a signal processing apparatus including 
means for acquiring second signals of a second dimension by projecting first signals 
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as real-world signals of a first dimension on a sensor and by detecting the mapped 
signals by the sensor, the first dimension being lower than the first dimension, and 
signal processing means for extracting the significant infonnation, buried by the 
projection fi-om the second signals, by perfonning signal processing which is based on 
the second signals. 

The present invention also provides a recording medimn having recorded 
thereon a computer-readable program, the program including a signal acquisition step 
of acquiring a second signal by projecting a first signal as a real world signal of a first 
dimension on a sensor and detecting the so-mapped first signal by the sensor, the 
signal being of a second dimension lower than the first dimension, and a signal 
processing step of perfonxiing signal processing based on the second signal to extract 
the significant infomiation buried by projection from the second signal. 

The present invention provides a signal processing apparatus including signal 
acquisition means for acquiring a second signal by detecting a first signal as a real 
world signal of a first dimension by a sensor, the signal being of a second dunension 
lower than the first dimension and containing distortion with respect to the first signal, 
and signal processing means for performing signal processing on the second signal for 
generating a third signal alleviated in distortion as compared to the second signal. 

The present invention also provides a signal processing apparatus for processing 
a predetennined number of detection signals acquired by a sensor made up of a 
predetermined number of detection elements having time integrating effects, the signal 



11 

processing apparatus including area specifying means for specifying a foreground area 
made up only of foreground object components constituting an foreground object, a 
background area made up only of background object components constituting a 
background object, and a mixed area mixed from the foreground object components 
and the background object components, mixed area detection means for detecting a 
mixing ratio of the foreground object components and the background object 
components at least in the mixed area, and separating means for separating the 
foreground object and the background object from each other based on the specified 
results by the area specifying means and the mixing ratio. 

The present invention also provides a signal processing method for processing 
a predetemiined number of detection signals acquired by a sensor made up of a 
predetermined number of detection elements having time integrating effects, the signal 
processing method including an area specifying step of specifying a foreground area, 
made up only of foreground object components constituting an foreground object, a 
background area made up only of background object components constituting a 
background object, and a mixed area mixed from the foreground object components 
and the background object components, a mixed area detection step of detecting a 
mixing ratio of the foreground object components and the background object 
components at least in the mixed area, and a separating step of separating the 
foreground object and the background object from each other based on the specified 
results by the area specifying means and the mixing ratio. 
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The present invention also provides a recording medium having a computer- 
readable program, recorded thereon, the computer-readable program including an area 
specifying step of specifying a foreground area, made up only of foreground object 
components constituting an foreground object, a background area made up only of 
background object components constituting a background object, and a mixed area 
mixed from the foreground object components and the background object components, 
a mixed area detection step of detecting a mixing ratio of the foreground object 
components and the background object components at least in the mixed area and a 
separating step of separating the foreground object and the background object from 
each other based on the specified results by the area specifying means and the mixing 
ratio. 

The present invention also provides a signal processing apparatus for processing 
a predetennined number of detection signals acquired by a sensor made up of a 
predetermined nmnber of detection elements having time integrating effects, the signal 
processing apparatus including area specifying means for specifying a foreground area, 
made up only of foreground object components constituting an foreground object, a 
background area made up only of background object components constituting a 
background object, and a mixed area mixed from the foreground object components 
and the background object components, and mixing ratio detecting means for detecting 
a mixing ratio between the foreground object components and the background object 
components at least in the mixed area based on the results specified by the area 
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specifying means. 

The present invention also provides a signal processing method for processing 
a predetermined number of detection signals acquired by a sensor made up of a 
predetennined nmnber of detection elements having time integrating effects, the signal 
processing method including an area specifying step of specifying a foreground area, 
made up only of foreground object components constituting an foreground object, a 
background area made up only of background object components constituting a 
background object, and a mixed area mixed from the foreground object components 
and the background object components, and a mixing ratio detecting step of detecting 
a mixing ratio between the foreground object components and the background object 
components at least in the mixed area based on the results specified by the area 
specifying means. 

The present invention also provides a recording medium having a computer- 
readable program recorded thereon, the signal processing method for processing a 
predetennined number of detection signals acquired by a sensor made up of a 
predetennined number of detection elements having time integrating effects, the 
computer-readable program including an area specifying step of specifying a 
foreground area, made up only of foreground object components constituting an 
foreground object, a background area made up only of background object components 
constituting a background object, and a mixed area mixed from the foreground object 
components and the background object component, and a mixing ratio detecting step 
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of detecting a mixing ratio between the foreground object components and the 
background object components at least n the mixed area based on the results specified 
by the area specifying means. 

The present invention also provides a signal processing apparatus for processing ' 
a predetermined number of detection signals acquired by a sensor made up of a 
predetennined number of detection elements having time integrating effects, the signal 
processing apparatus including mixing ratio detecting means for detecting a mixing 
ratio of foreground object components and background object components in a mixed 
area in which said foreground object components constituting a foreground object and 
said background object components constituting a background object are mixed, and 
separating means for separating the foreground object and the background object from 
each other based on the mixing ratio. 

The present invention provides a signal processing method for processing a 
predetermined number of detection signals acquired by a sensor made up of a 
predetennined number of detection elements having time integrating effects, the signal 
processing method including a mixing ratio detecting step of detecting a mixing ratio 
of foreground object components and background object components in a mixed area 
in which said foreground object components constituting a foreground object and said 
background object components constituting a background object are mixed, and a 
separating step of separating the foreground object and the background object from 
each other based on the mixing ratio. 
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The present invention also provides a recording medium having recorded 
thereon a computer-readable program for processing a predetermined number of 
detection signals acquired by a sensor made up of a predetennined number of 
detection elements having time integrating effects, the computer-readable program 
including a mixing ratio detecting step of detecting a mixing ratio of foreground 
object components and background object components in a mixed area in which said 
foreground object components constituting a foreground object and said background 
object components constituting a background object, and a separating step of 
separating the foreground object and the background object from each other based on 
the mixing ratio. 

Brief Description of the Drawings 

Fig. 1 illustrates the principle of the present invention. 

Fig.2 is a block diagram showing an illustrative structure of a system embodying 
the present invention. 

Fig. 3 is a block diagram showing an illustrative structure of a signal processor 
of Fig.2. 

Fig.4 is a flowchart for illustrating the operation of the system of Fig.2. 
Fig. 5 illustrates a typical picture acquired at step SI of Fig.4. 
Fig. 6 illustrates pixel values of a mixed area. 

Fig. 7 illustrates the result of subtracting picture components of the background 
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in domains Dl to D3 of Fig.6. 

Fig. 8 illustrates the structure of motion blurring. 

Fig. 9 is a flowchart for illustrating another typical processing of the system of 

Fig.2. 

Fig. 10 is a block diagram showing a signal processor 12. 

Fig. 1 1 illustrates the photographing by a sensor. 

Fig. 12 illustrates pixel arrangement. 

Fig. 13 illustrates the operation of a detection device. 

Fig. 14 illustrates a picture obtained on imaging an object corresponding to the 
moving foreground and an object corresponding to a stationary background. 

Fig. 1 5 illustrates a background area, a foreground area, a mixed area, a covered 
background area and an uncovered background area.. 

Fig. 16 is a diagraiumatic view showing pixel values of pixels arranged in a 
neighboring relation to another in a row in a picture obtained on imaging an object 
corresponding to the stationary foreground and a picture obtained on imaging an 
object corresponding to the stationary background, with the pixel values extended in 
the time axis direction. 

Fig. 17 is a diagrammatic view showing pixel values extended in the time axis 
direction, with the time period corresponding to the shutter time shown split. 

Fig. 18 is a diagrammatic view showing pixel values extended in the time axis 
direction, with the time period corresponding to the shutter time shown split. 
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Fig. 19 is a diagrammatic view showing pixel values extended in the time axis 
direction, with the time period corresponding to the shutter time shown split. 

Fig.20 shows extracted pixels of a foreground area, a background area and a 
mixed area. 

Fig.21 shows the relation of correspondence between pixels and a model 
obtained on expanding the pixel values in the time axis direction. 

Fig.22 is a diagraimnatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.23 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig. 24 is a diagraimnatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig. 2 5 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.26 is a diagraimnatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.27 is a flowchart for illustrating the processing for adjusting the amount of 
the motion blurring. 

Fig.28 is a block diagram showing an illustrative structure of an area specifying 
unit 103. 

Fig.29 illustrates a picture as an object corresponding to the foreground is being 
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moved. 

Fig.30 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.3 1 is a diagraimnatic view showing pixel values developed in the time axis - 
direction and showing the time period corresponding to the shutter period shown split. 

Fig. 32 is a diagraimnatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.33 illustrates a condition for areal decision. 

Figs.34A, 34B, 34C and 34D illustrate the results of identification of areas of 
the area specifying unit 103. 

Fig.35 illustrates the results of identification of areas of the area specifying unit 

103. 

Fig.36 is a flowchart for illustrating the processing for areal identification. 
Fig. 3 7 is a block diagram showing an illustrative structure of a mixing ratio 
calculating unit 104. 

Fig. 3 8 shows a typical ideal mixing ratio a. 

Fig.39 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.40 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown spHt. 

Fig.4 1 illustrates the approximation exploiting the correlation of the foreground 
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components. 

Fig.42 illustrates the relation between C, N and P. 

Fig.43 is a block diagram showing the structure of an estimated mixing ratio 
processor 201. 

Fig. 44 shows a typical estknated mixing ratio. 

Fig.45 is a block diagram showing a modified structure of the mixing ratio 
calculating unit 104 . 

Fig.46 is a flowchart for illustrating the processing for calculating the estimated 
mixing ratio. 

Fig.47 is a flowchart for illustrating the processing for the operation of the 
estimated mixing ratio. 

Fig.48 is a block diagram showing an illustrative structure of a 
foreground/background separating unit 105. 

Figs.49A and 49B show an input picture, a foreground component picture and 
a background component picture. 

Fig.50 is a diagrammatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig. 5 1 is a diagraimnatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 

Fig.52 is a diagraimnatic view showing pixel values developed in the time axis 
direction and showing the time period corresponding to the shutter period shown split. 
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Fig. 53 is a block diagram showing an illustrative structure of a separator 25 1. 

Figs.54Aand 54B illustrate typical examples of a foreground component picture 
and a background component picture as separated from each other. 

Fig.55 is a flowchart for illustrating the processing for separating the foreground ' 
and the background from each other. 

Fig.56 is a block diagram showing an illustrative structure of amotion blurring 
adjustment unit 106. 

Fig. 57 illustrating a processing unit. 

Fig. 5 8 is a diagrammatic view showing pixel values of a foreground component 
picture developed in the time axis direction and showing the time period corresponding 
to the shutter period shown split. 

Fig. 59 is a diagrammatic view showing pixel values of a foreground component 
picture developed in the time axis direction and showing the time period corresponding 
to the shutter period shown split. 

Fig.60 is a diagrammatic view showing pixel values of a foreground component 
picture developed in the time axis direction and showingthe time period corresponding 
to the shutter period shown split. 

Fig. 6 1 is a diagraimnatic view showing pixel values of a foreground component 
picture developed in the time axis direction and showing the time period corresponding 
to the shutter period shovra split. 

Fig.62 shows a modified structure of the motion blurring adjustment unit 106. 
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Fig. 63 is a flowchart for illustrating the processing for adjusting the amount of 
motion blurring contained in the foreground component picture 

Fig. 64 is a block diagram showing a modified structure of the function of a 
signal processor 12. 

Fig. 6 5 shows the structure of a synthesis unit 371. 

Fig. 6 6 is a block diagram showing another modified structure of the function 
of the signal processor 12. 

Fig. 67 is a block diagram showing the structure of a mixing ratio calculating 
unit 401. 

Fig.68 is a block diagram showing the structure of a foreground/background 
separating unit 402. 

Fig.69 is a block diagram showing a further modified structure of the fimction 
of the signal processor 12. 

Fig.70 illustrates the structure of a synthesis unit 431. 

Fig.71 shows another illustrative structure of a signal processing apparatus 
according to the present invention. 

Fig. 72 is a flowchart for illustrating the processing for adjusting the amount of 
motion blurring by a signal processor 452. 

Fig.73 shows an illustrative structure of a signal processing apparatus according 
to the present invention. 

Fig. 74 shows a structure of a pressure area sensor 501. 
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Fig.75 illustrates the load applied to the pressure area sensor 501. 

Fig.76 illustrates typical weight data output by the pressure area sensor 501. 

Fig. 77 is a flowchart for illustrating the load calculating processing executed by 
a signal processor 502. 

Fig. 7 8 is a block diagram showing the structure of generating a picture having 
an increased number of pixels per frame, as another fimction of the signal processor 
12. 

Fig. 79 illustrates pixel arrangement and an area corresponding to a pixel 
doubled in horizontal density. 

Fig. 80 illustrates a picture component of a picture corresponding to light input 
to areas A to r. 

Figs. 81 A, 8 IB, 81C and 8 ID illustrates calculation of picture components 
corresponding to two areas of a pixel. 

Fig. 82 shows a typical input picture. 
Fig. 83 shows a typical double horizontal density picture. 
Fig. 84 shows a typical double vertical density picture. 
Fig. 8 5 shows a double density picture. 

Fig. 8 6 is a flowchart for illustrating the processing for generating a double 
density picture by a signal processor 12 shown in Fig.78. 



Best mode for Carrying out the Invention 
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Fig. 1 shows the principle of the present invention. As may be seen in Fig. 1, a 
first signal, as the infonnation of a real world 1 having the spatial axis and the 
temporal axis, is acquired by a sensor 2, and is made into data. A detection signal, as 
data 3 acquired by the sensor 2, is the infonnation obtained on projecting the 
information of the real world 1 on a tune space of a lower dimension than in the real 
world 1 . Therefore, the infonnation, resulting from the projection, contains distortion 
ascribable to projection. Stated differently, the data 3, output by the sensor 2, is 
distorted relative to the infonnation of the real world 1. Moreover, the data 3, thus 
distorted as a result of projection, also includes the significant infonnation usable or 
correcting the distortion. 

Thus, according to the present invention, the data output by the sensor 2 is 
processed by a signal processor 4, whereby the distortion is removed, reduced or 
adjusted. Alternatively, the data output by the sensor 2 is processed by the signal 
processor 4 to extract the significant infonnation. 

Fig.2 shows an illustrative structure of a signal processing apparatus according 
to the present invention The sensor 1 is comprised e.g., of a video camera, which 
photographs a picture of the real world to output the resulting picture data to the signal 
processor 12. The signal processor is comprised e.g., of a personal computer for 
processing the data input from the sensor 11, adjusting the amount of distortion 
produced by projection, specifying an area containing the significant infonnation 
buried by the projection, extracting the significant infonnation from a specified area 
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and for processing the input data based on the extracted significant infonnation. 

The significant infonnation may, for example, be a mixing ratio, as later 
explained. 

Meanwhile, the infonnation indicating an area containing the significant 
information buried by the projection may also be deemed to be the significant 
infonnation. Here, the areal infonnation, as later explained, corresponds to the 
significant infonnation. 

The signal processor 12 is configured as shown for example in Fig. 3. A CPU 
(central processing unit) 2 1 executes various processing operations in accordance with 
a program stored in a ROM (read-only memory) 22 or in a storage unit 28. In a RAM 
(random access memory) 23, the program executed by the CPU 2 1 or data are stored 
as necessary. The CPU 21, ROM 22 and the RAM 23 are interconnected over a bus 
24. 

To the CPU 21 is connected an input/output interface 25 over a bus 24. To the 
input/output interface 25 are connected an input unit 26, comprised of a keyboard, a 
mouse and a microphone, and an output unit 27, comprised of a display and a speaker. 
The CPU 21 executes various processing operations responsive to commands input 
from the input unit 26. The CPU 21 outputs a picture, speech and so forth, obtained 
on processing, to the output unit 27. 

The storage unit 28, connected to the input/output interface 25, is constituted 
e.g., by a hard disc, for storing the program executed by the CPU 2 1 and a variety of 
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data. A communication unit 29 communicate with external equipment over a network, 
such as Internet. In the present embodiment, the communication unit 29 operates for 
acquiring an output of the sensor 1 1. 

The program may also be acquired over the communication unit 29 for storage ' 
in the storage unit 28. 

A driver 30 connected to the input/output interface 25 drives a magnetic disc 
51, an optical disc 52, a magneto-optical disc 53 or a semiconductor memory 54, to 
acquire the program and data recorded therein, when these devices are connected 
thereto. The program and the data, thus acquired, are transferred to the storage unit 
28, as necessary, for storage therein. 

Referring to the flowchart of Fig.4, the operation perforaied by the signal 
processing apparatus based on the program stored in the storage unit 28, is explained. 
First, at step SI, a picture of an object, acquired by the sensor 11, is acquired through 
e.g., the communication unit 29. The CPU 21 of the signal processor 12 sends the 
acquired picture data to the storage unit 28 for storage therein. 

Fig. 5 shows a picture associated with the so-acquired picture data. The picture, 
shown in this embodiment, is comprised of a foreground 62 arranged ahead of a 
background 6 1 . The foreground here is a toy plane moving at a predetermined speed 
ahead of the still background 61 towards right in the drawing. The result is that the 
picture of the foreground 62 is a picture subjected to so-called motion blurring. 
Conversely, the picture of the background 6 1 is stationary and hence is a clear picture 
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free of motion blurring. A mixed area 63 is a picture comprised of a mixture of an 
object which is the background 61 and an object which is the foreground 62. 

Then, at step S2, the CPU 21 detects the mixed area of the objects. In the 
embodiment of Fig. 5, the mixed area 63 is detected as an area of the mixture of the 
two objects. 

The CPU 2 1 at step S3 decides whether or not the objects are mixed. If the 
objects are not mixed, that is if there is no mixed area 63, the picture is not what is to 
be processed by the present infonnation processing apparatus and hence the 
processing is finished. 

If conversely a decision is made at step S3 that the objects are mixed, the CPU 
2 1 proceeds to step S4 to find the object mixing ratio in the detected mixed area. The 
mixing ratio may be found by finding the motion vector of the foreground 2 1 relative 
to the background 61 and by fitting, from the motion vector, so that the mixing ratio 
in the mixed area 63 will be changed in a range from 0 to 1. At step S5, the CPU 21 
perforais the processing of separating the objects in the mixed area 63 where plural 
objects are mixed together, based on the so-found mixing ratio. 

The above-described processing is explained in frirther detail, taking a picture 
of Fig. 5 as an example. If picture data on one line of a portion 63 A on the right end 
of the mixed area 63 of Fig.5 is plotted, the result is as shown in Fig.6, in which the 
abscissa denotes X-coordinates (coordinates in the horizontal direction in Fig.5) and 
the ordinate denotes pixel values on the X-coordinates. 
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A curve LI denotes pixel values on a line of a first timing, whilst a curve L2 
denotes pixel values on another line of the next timing. Similarly, curves Light 
reflecting layer 3 and L4 denote pixel values of lines of the sequentially consecutive 
timings. Stated differently, Fig.6 shows changes in the pixel values on associated lines 
at the four consecutive timings. 

The curve Li shows the state in the first timing in which state the foreground 62 
has not yet been imaged. So, the curve LI represents pixels of the foreground 61. 

On the curve LI, the pixel value is approximately 75 in the vicinity of the X- 
coordinate 140, and is increased to approximately 130 at the X-coordinate 145. The 
pixel value then is lowered and is approximately 120 in the vicinity of the X- 
coordinate 149. As the X-coordinate is increased, the pixel value is again increased 
and reaches approximately 160 in the vicinity of the X-coordinate 154. The pixel 
value then is again lowered and reaches approximately 130 in the vicinity of the X- 
coordinate 162. Then, in the vicinity of the X-coordinate of 165, the pixel value is 
approximately 180 and, in the vicinity of the X-coordinate of 170, the pixel value is 
again lowered to approximately 125 . Then, in the vicinity of the X-coordinate of 1 72, 
the pixel value is increased to approximately 175 and, in the vicinity of the X- 
coordinate of 178, the pixel value is lowered to approximately 60. Subsequently, the 
pixel value is slightly fluctuated between 60 and 80 in a domain of the X-coordinates 
of from 178 to 195. In the X-coordinates on the further right side of approximately 
195, the pixel value is again increased to approximately 160. 
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As for the curve L2 of the next frame, the pixel value is constant at 
approximately 200 up to the pixel value of 145. The pixel value then is gradually 
lowered in a range from the X-coordinate of 145 to the Y-coordinate of 160, at which 
Y-coordinate value the pixel value is approximately 125. The curve then undergoes 
changes in a manner similar to those of the curve LI. 

The pixel value of the curve Light reflecting layer 3 is substantially constant at 
200 up to the vicinity of the X-coordinate 158 and is then lowered to approximately 
164 at the X-coordinate 164, after which it is increased to approximately 190. The 
curve then undergoes changes in a manner similar to those of the curve LI. 

The pixel value of the curve L4 is constant at approximately 200 from the 
vicinity of the X-coordinate of 140 up to the vicinity of the X-coordinate 170, and is 
abruptly lowered from the vicinity of the X-coordinate of 170 up to the vicinity of the 
X-coordinate 1 80, with the pixel value in the vicinity of the X-coordinate of 1 70 being 
approximately 70. The curve then undergoes changes in a manner similar to those of 
the curve LI. 

These changes in the pixel values of the curves L2 to L4 are ascribable to the 
fact that, while the picture of only the background 61 exists in the state of the curve 
LI, the picture of the foreground 62 is gradually increased with the movement of the 
picture of the foreground 62, that is with lapse of time. 

Specifically, as may be seen from comparison of the curve LI and the curve L2 
of the next following timing, the values of the curves L2 to L4 are substantially equal 
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in values up to the vicinity of the X-coordinate of 147. Beginning from the vicinity of 
the X-coordinate 147, the values of the curve L2 differ from those of the curves Light 
reflecting layer 3, L4,becoming equal to the values of the curve LI in vicinity of the 
X-coordinate 159. Subsequently, the pixel values of the curve L2 are approximately 
equal to those in the curve LI. That is, the values of the curve L2 in an area Rl 
corresponding to a domain Dl from an X-coordinate 146 to an X-coordinate 159 
indicate that the foremost part of the foreground 62 has been moved from the left end 
to the right end of the domain Dl during one unit period. 

Similarly, the pixel values of the curve Light reflecting layer 3 of the next 
timing in an area Rigid substrate 2 corresponding to a domain D2 from an X- 
coordinate 159 to an X-coordinate 172 indicate that the foremost part of the 
foreground 62 has been moved in the interim. The pixel values of the curve L4 of the 
next timing in an area R3 corresponding to a domain D3 from the X-coordinate 172 
to an X-coordinate 184 indicate that the foremost part of the foreground 62 has been 
moved in the interim. 

So, if the pixel values of the curve LI, weighted on the basis of a mixing ratio 
of the foreground 62 to the background 6 1, are subtracted from the pixel values of the 
curve L2, a curve Lll shown in Fig.7 is obtained. This curve Lll, tantamount to 
subtraction of the values corresponding to background 61 from the pixels of the 
foreground 62 in the mixed area 63, represents a picture of the foreground on the 
background having the pixel value of 0. Meanwhile, in Fig.7, the abscissa and the 



30 

ordinate denote the position and the pixel values of the extracted foreground, 
respectively. As for the position, the left and the right end correspond to the left and 
right ends in the domain Dl in Fig. 6, respectively. 

Similarly, if, in the domain D2 of Fig.6, the pixel values of the curve LI, 
weighted by the mixing ratio, are subtracted from the pixel values of the curve Light 
reflecting layer 3, a curve L12 in Fig.7 is obtained, whereas, if, in the domain D3 of 
Fig.6, the pixel values of the curve LI, weighted by the mixing ratio, are subtracted 
from the curve L4, a curve LIS in Fig.7 is obtained. The curves L12, LIS are 
substantially coincident with the curve Lll , as shown in Fig.7. This indicates that the 
foreground 62 is moving at an approximately equal speed during the three timing unit 
periods, and that the black background, that is the foreground pixel values on the 
background having the pixel value of 0, has been obtained correctly by weighted 
subtraction. 

If the above-described operation is explained in connection with pixels by 
referring to Fig. 8, in which the abscissa denotes the X-coordinate of a portion 63 A, 
with the ordinate denoting the time axis directing from above towards below. Since 
the amount of movement is 5 in the present embodiment, light exposure is made within 
the time interval of tl to t5 (v^fithin the shutter time). In Fig. 8, bl to bf denote pixel 
values of the respective pixels of the background 6 1 and Al to A6 denote pixel values 
of the foreground 62. 

That is, the pixels Al to A6 of the foreground 62 appear at the positions of the 
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pixels b3 to b8 of the background 61, with the pixels Al to A6 of the foreground 62 
moving rightwards at the timing t2 by one pixel, that is to the position of the pixels b4 
to b9 of the background 61. 

In similar manner, the pixels Al to A6 of the foreground 62 are sequentially 
moved rightwards at a pitch of one pixel as time elapses from timing t3 to timing t5. 
In this case, the pixel values yl to yf, obtained on averaging the pixels of the 
respective lines of the timings tl to t5, constitute pixels obtained on imaging, that is 
pixels exhibiting motion blurring, with the values being represented by the following 
equations: 

y, = --(^/i + C/2 + ^3) + ^-^5 ...(3) 

y, = -• (a, +a2 + a^ + a^)+ --b, _ 
yj = -■(a, + 0^ + aj + a, + a,) _ 
1 

y, ^--{a. + a. + a. + a. + a.ia,) ... 

y9 = l-(a, + a, + a, + a,) + ^-b, ... 
1 2 

y,=Y(^s+^e) + yb, ...(9) 

y. = -,-.^^-K ...(10) 

Meanwhile, yl, y2, yd, ye and yf are equal to background pixels bl, b2, bd, be 
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and bf, respectively. 

If pixels bl to bf of the background are removed, the background 61 and the 
foreground 62 in the mixed area 63 can be separated from each other. That is, plural 
objects can be separated from one another. Moreover, the background pixels b 1 to bf 
can be found by solving the above equations, using, for example, the least square 
method, by assuming the background pixels b 1 to bf to be known such as by 
employing the pixel values of the fore and aft shutter time (frame). This gives a 
foreground picture freed of the motion blurring. In this manner, distortion caused by 
projection in the infomiation of the real world can be reduced to create a clear picture 
such as by processing for resolution creation. 

In Fig.4, it is the deterministic processing that is executed, that is, the previous 
processing is used as basis and the next following processing is executed on the 
assmnption that the result of the previous processing is just. Alternatively, statistic 
processing is also possible, as now explained with reference to illustrative processing 
shown in Fig.9. 

Specifically, when carrying out the statistic processing, the CPU 21 acquires 
picture data at step S21. This processing is similar to that performed at step SI in 
Fig.4. 

Next, st step S22, the CPU 21 performs the processing of fmding the mixing 
ratio of the foreground an the background from the picture data obtained at step S21 . 
At step 823, the CPU 2 1 executes the processing of separating the foreground and the 
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background based on the mixing ratio found at step S22. 

If the statistic processing is used, the processing of deciding whether or not the 
boundary of an object exists, such as that at step S23 of Fig.4, is unnecessary, thus 
enabhng the foreground and the background to be separated from each other more 
expeditiously. 

The foregoing shows the manner as to how a clear picture of the foreground 62 
can be separated and extracted from the motion-blurred picture obtained on 
photographing a picture of the foreground 62 moving ahead of the background 61. 

A more specified embodiment of a signal processing apparatus for identifying 
an area having the significant information buried therein or extracting the so-buried 
significant infonnation from data acquired from the sensor by the detenninistic 
processing is now explained. In the following embodiment, a CCD line sensor or a 
CCR area sensor corresponds to the sensor, while the areal infonnation or the mixing 
ratio corresponds to the significant information and the mixing of the foreground and 
the background or the motion blurring corresponds to distortion. 

Fig. 10 is a block diagram showing the signal processor 12. 

Meanwhile, it does not matter whether the respective ftinctions of the signal 
processor 12 are to be implemented by hardware or by software. That is, the block 
diagrams of the present specification may be deemed to be a hardware block diagram 
or a fimctional software block diagram. 

It is noted that the motion blurring means distortion contained in a moving 
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object, which distortion is produced by the movement of an object in the real world 
being imaged and by imaging characteristics proper to the sensor 1 1 . 

In the present specification, the picture corresponding to an object in the real 
world is called a picture object. 

An input picture, supplied to the signal processor 12, is furnished to an object 
extraction unit 101, an area specifying unit 103, ainixingratio calculating unit 104 and 
a foreground background separating unit 105. 

The object extraction unit 101 roughly extracts a picture object corresponding 
to a foreground object contained in the input picture to send the extracted picture 
object to a motion detection unit 102. The object extraction unit 101 detects the 
contour of the picture object corresponding to the foreground object contained in the 
input picture to roughly extract the picture object corresponding to the foreground 
object. 

The object extraction unit 101 roughly extracts the picture object corresponding 
to the foreground object contained in the input picture to route the extracted picture 
object to the motion detection unit 102. The object extraction unit 101 roughly 
extracts the picture object corresponding to the background object, based on the 
difference between the input picture and the picture object corresponding to the 
extracted foreground object. 

It is also possible for the object extraction unit 101 to roughly extract the 
picture object corresponding to the foreground object and the picture object 
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corresponding to the background object based on the difference between the 
background picture stored in an internal background memory and the input picture. 

The motion detection unit 102 computes the motion vector of the picture object 
corresponding to the roughly extracted foreground, by techniques such as block 
matching method, gradient method, phase correlation method or the Pel-Recursive 
method, to route the motion vector so calculated and the position infonnation of the 
motion vector (the information specifying the position of the pixel corresponding to 
the motion vector) to the motion blurring adjustment unit 106. 

In the motion vector, output by the motion detection unit 1 02, there is contained 
the infonnation corresponding to a movement quantity v. 

It is also possible for the motion detection unit 102 to output the picture object 
based motion vector, along with the pixel position information specifying the pixels 
for the picture object, to the motion blurring adjustment unit 106. 

The movement quantity v is a value for representing position changes of picture 
corresponding to a moving object in ternis of a pixel-to-pixel interval as unit. For 
example, if a picture of an object corresponding to the foreground is moved so as to 
be displayed at a position offset by four pixels in a frame with respect to a directly 
previous frame, the movement quantity v of the object corresponding to the 
foreground is 4. 

Meanwhile, the object extraction unit 101 and the motion detection unit 1 02 are 
used when the quantity of the motion blurring associated with a moving object is 
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adjusted in the motion blurring adjustment unit 106. 

The area specifying unit 103 sends the infonnation specifying each pixel of an 
input picture to one of the foreground area, a background area or a mixed area and for 
indicating to which of the foreground area, background area and the mixed area belong 
the pixels, from pixel to pixel, to the mixing ratio calculating unit 104, 
foreground^ackground separating unit 1 05 and to the motion blurring adjustment unit 
106. The aforementioned information is referred to below as the area infonnation. 

The mixing ratio calculating unit 104 calculates the mixing ratio for pixels 
contained in the mixed area 63, based on the input picture and the area infonnation 
supplied from the area specifying unit 103, to route the so-calculated mixing ratio to 
the foreground/background separating unit 105. This mixingratio is referred to below 
as a mixing ratio a. 

The mixing ratio a indicates the proportion in the pixel value of the components 
of a picture corresponding to the background object, as indicated by an equation (13) 
to be described later. These components are also referred to below as the background 
components. 

The foreground^ackground separating unit 105 separates the input picture into 
a foreground component picture, made up only of a picture component associated with 
the foreground, also referred to below as foreground components, and a background 
component picture, composed only of background components, based on the area 
mfonnation supplied from the area specifying unit 103, and on the mixing ratio Ct 
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with the mixing ratio a to route the resulting product to the switch 322, which is 
closed when the pixel of the fraine #n (corresponding to the pixel of the frame #n+l) 
supplied from the frame memory 30 1 is in the uncovered background area, to route the 
pixel value multiplied with the mixing ratio 0£ sent from the multiplier 321 to an 
operating unit 322 and to the synthesis unit 314. The pixel value of the pixel of the 
frame #n+l from the switch 322, multiplied by the mixing ratio a, is equal to the 
background component of the pixel value of the corresponding pixel of the frame #n. 

An operating unit 323 subtracts the background component supplied from the 
switch 322 from the pixel value of the pixel of the frame #n supplied from the frame 
memory 301 to find the foreground component. The operating unit 323 routes the 
foreground component of the pixel of the frame #n, belonging to the uncovered 
background area, to the synthesis unit 313. 

The covered area processor 3 1 2 includes a multiplier 33 1 which multiplies the 
pixel value of the pixel of the frame #n* 1 supplied from the frame memory 301 with 
the mixing ratio a to route the resulting product to the switch 322, which is closed 
when the pixel of the frame #n (corresponding to the pixel of the frame #n- 1 ) supplied 
from the frame memory 30 1 is in the covered background area, to route the pixel value 
multiphed with the mixing ratio a sent from the multiplier 33 1 to an operating unit 333 
and to the synthesis unit 314. The pixel value of the pixel of the frame #n' 1 from the 
switch 332, multiplied by the mixing ratio a, is equal to the background component 
of the pixel value of the corresponding pixel of the frame #n. 
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if so desired. 

The selection unit 107 selects one of the foreground component picture supplied 
from the foreground^ackground separating unit 105 and the foreground component 
picture from the motion blurring adjustment unit 106, adjusted as to the motion 
blurring quantity, to output the selected foreground component picture. 

Referring to Figs. 1 1 to 26, an input picture sent to the signal processor 12 is 
explained. 

Fig.l 1 illustrates imaging by a sensor 1 1 constituted by a CCD video camera 
provided with a CCD (charge coupled device) which is a solid state imaging device. 
An object corresponding to the foreground in the real world is moved between the 
object of the background in the real world an the sensor 1 1 e.g., horizontally from left 
to right. 

The sensor 1 1 images an object corresponding to the foreground along with the 
object corresponding to the background. The sensor 1 1 outputs the photographed 
picture on the frame basis. For example, the sensor 1 1 outputs a picture of 30 frames 
per sec. The exposure time of the sensor 1 1 may be set to 1/30 sec. The exposure time 
is the time which elapses since the start of conversion of light input to the sensor 1 1 
into electrical charges until the end of the conversion of the input light into electrical 
charges. This exposure time is sometimes referred to below as the shutter time. 

Referring to Fig. 1 2, showing pixel arrangement, A to I denote individual pixels. 
The pixels are arranged in a plane corresponding to a picture. A detection element 
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associated with one pixel is arranged on the sensor 11. When the sensor 11 
photographs a picture, one detection element outputs a pixel value associated with one 
pixel belonging to the picture. For example, the position of the detection device along 
the X-direction corresponds to the position on the picture in the transverse direction, 
whilst that along the Y-direction corresponds to the position on the picture in the 
longitudinal direction. 

Referring to Fig. 13, a detection device, such as the CCD, converts the input 
light into electrical charges, during the time corresponding to the shutter time, to store 
the as-converted electrical charges. The quantity of the electrical charges is 
approximately equal to the intensity of the input light and to the time during which the 
light is input. The detection device sums the electrical charges, converted from the 
input light, to the electrical charges, already stored, during the time corresponding to 
the shutter time. That is, the detection device integrates the input light during the time 
corresponding to the shutter time to accumulate electrical charges in an amount 
corresponding to the integrated light. The detection device is said to have an 
integrating effect with respect to time. 

The charges accumulated in the detection device are converted into an electrical 
voltage by a circuit, not shown. The voltage, in turn, is converted into a pixel value, 
such as digital data, which is output. So, the individual pixels, output by the sensor 1 1 , 
are of a value mapped to a one-dimensional space, which is the result of integration 
with respect to the shutter time of a spatially extended portion of an object 
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corresponding to the foreground or the background. 

By such accumulating operation of the sensor 11, the signal processor 12 
extracts the significant information buried in the output signal, such as the mixing ratio 
a. The signal processor 12 adjusts the quantity of distortion caused by the mixing of 
no other than the foreground picture object, for example, the quantity of the motion 
blurring. The signal processor 12 also adjusts the quantity of the distortion produced 
by the mixing of the foreground picture object with the background picture object. 

Fig. 14 illustrates a picture obtained on imaging an object corresponding to a 
moving foreground and an object corresponding to a still background. Fig. 14A shows 
a picture obtained on imaging an object corresponding to the moving foreground and 
an object corresponding to the still background. In an embodiment shown in Fig. 1 4 A, 
the object corresponding to the foreground is moving horizontally fi-om left towards 
right relative to the picture. 

Fig. 14B is a diagrammatic view showing pixel values, corresponding to a line 
of the picture shown in Fig. 14 A, as extended along the time axis. The transverse 
direction of Fig.l4B corresponds to the spatial direction X of Fig.l4A. 

The pixels of the background area are constituted solely by the background 
components, that is components of a picture corresponding to a background object. 
The pixels of the foreground area are constituted solely by the foreground 
components, that is components of a picture corresponding to a foreground. 

The pixels of the mixed area are constituted from the background and 
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foreground components. The mixed area, the pixel values of which are constituted 
from the background components and the foreground components, may be said to be 
a distorted area. The mixed area is further classified into a covered background area 
and an uncovered background area. 

The covered background area is a portion of the mixed area in register with the 
foremost part along the proceeding direction of the foreground and is an area in which 
the background component is hidden by the foreground with lapse of time. 

On the other hand, the uncovered background area is a portion of the mixed 
area in register with the rear part along the proceeding direction of the foreground and 
is an area in which the background component presents itself with lapse of time. 

A picture comprised of the foreground area, background area, a covered 
background area or the uncovered background area is input as an input picture to the 
area specifying unit 103, mixing ratio calculating unit 104 and to the 
foreground/background separating unit 105. 

Fig. 15 illustrates the background area, foreground area, mixed area, covered 
background area and the uncovered background area, as described above. In relation 
to the picture shown in Fig. 14, the background area is a still portion, the foreground 
area is a moving portion, the covered background area of the mixed area is an area 
where the picture is changed from the background to the foreground, and the 
uncovered background area of the mixed area is an area where the picture is changed 
from the foreground to the background. 
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Fig. 16 diagrammatically shows pixel values of neighboring pixels in a row in 
a photographed picture of an object corresponding to a still foreground and an object 
corresponding to a sill background, with the pixel values shown developed along the 
temporal axis direction. As the neighboring pixels, arranged in a row, it is possible to 
select pixels arranged on a line of a picture. 

The pixel values of FOl to F04, shown in Fig. 16, are those of pixels of the 
object of the still foreground. The pixel values of BOl to B04, shown in Fig. 16, are 
those of pixels of the object of the still background. 

In Fig. 16, time elapses from above towards below. The position of an upper 
side of a rectangle in Fig. 16 corresponds to the time the sensor 1 1 begins converting 
the input light into electrical charges, while that of the rectangle in Fig. 1 6 corresponds 
to the time the sensor 1 1 finishes the conversion of the input light into electrical 
charges. That is, the distance from the upper to the lower sides of the rectangle of 
Fig. 16 corresponds to the shutter time. 

In the following description, it is assumed that the shutter time is equal to the 
frame interval. 

The transverse direction in Fig. 16 corresponds to the spatial direction X, 
explained with reference to Fig. 14. More specifically, the distance from the left side 
of a rectangle "FOl" to the right side of a rectangle "B04" in Fig.l6 is eight times the 
pixel pitch, that is the span of the eight consecutive pixels. 

If the foreground and the background object are still, the light input to the 
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sensor 1 1 is not changed during the time corresponding to the shutter time. 

The time span corresponding to the shutter time is spHt into two or more equal 
time periods. For example, if the number of times of the virtual splitting is four, the 
diagram of Fig. 16 may be represented as the diagram of Fig. 17. The number of times 
of the virtual splitting is set in association with e.g., the movement quantity v in the 
shutter time of the object corresponding to the foreground. For example, if the 
movement quantity v is four, the number of times of the virtual splitting is 4, with the 
time span corresponding to the shutter time being then split into four. 

The uppennost row in the drawing corresponds to the first split time period 
since the time of shutter opening. The second row corresponds to the second split time 
period since the time of shutter opening. The third row corresponds to the third split 
time period since the time of shutter opening, whilst the fourth row corresponds to the 
fourth split time period since the time of shutter opening. 

The shutter rime split in association with the movement quantity v is also called 
the shutter time/v hereinbelow. 

When the object corresponding to the foreground is at a standstill, the light 
input to the sensor 11 is not changed. So, the foreground component FOl/v is equal 
to the pixel value FOl divided by the number of times of the virtual splitting. 
Similarly, when the object corresponding to the foreground is at a standstill, the 
foreground component F02/v is equal to the pixel value F02 divided by the number of 
times of the virtual splitting, whilst the foreground component F03/v is equal to the 
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pixel value F03 divided by the number of times of the virtual splitting and the 
foreground component F04/v is equal to the pixel value F04 divided by the number of 
times of the virtual splitting. 

When the object corresponding to the background is at a standstill, the light 
incident on the sensor 11 is not changed. So, the background component BOl/v is 
equal to the pixel value BOl divided by the number of times of the virtual splitting. 
Similarly, when the object corresponding to the background is at a standstill, the 
background component B02/v is equal to the pixel value B02 divided by the number 
of times of the virtual splitting, whilst the background component B03/v is equal to the 
pixel value BOS divided by the number of times of the virtual splitting and the 
background component B04/v is equal to the pixel value B04 divided by the number 
of times of the virtual splitting. 

That is, when the object corresponding to the foreground is at a standstill, the 
light corresponding to the foreground input to the sensor 11 during the time 
corresponding to the shutter time remains unchanged. So, the first foreground 
component FOlv, corresponding to the shutter time/v, as fi-omthe shutter opening, the 
second foreground component FOlv, corresponding to the shutter time/v, as from the 
shutter opening, the third foreground component FOlv, corresponding to the shutter 
trnie/v, as from the shutter opening and the fourth foreground component FOlv, 
corresponding to the shutter time/v, as from the shutter opening, are of equal values. 
The above for FOl/v holds for F02/v to F04/v as well. 
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When the object corresponding to the background is at a standstill, the Hght 
corresponding to the background object input to the sensor 11 during the time 
corresponding to the shutter time remains unchanged. So, the first background 
component BO 1 v, corresponding to the shutter time/v, as from the shutter opening, the' 
second background component BO 1 v, corresponding to the shutter time/v, as from the 
shutter opening, the third background component BO 1 v, corresponding to the shutter 
time/v, as from the shutter opening and the fourth background component BOIv, 
corresponding to the shutter time/v, as from the shutter opening, are of equal values. 
The above for BOl/v holds for B02/v to B04/v as well. 

In the following description, it is assumed that the object corresponding to the 
foreground is moving, with the object corresponding to the background being at a 
standstill. 

Fig. 18 diagrammatically shows pixel values of pixels arranged on a line 
including the covered background area when the object corresponding to the 
foreground is moving towards right in the drawing, with the pixel values being shown 
developed in the time axis direction. In Fig. 18, the movement quantity v of the 
foreground is 4. Since one frame is of short duration, it may be assumed that the 
object corresponding to the foreground is a rigid body and moving at an equal speed. 
In Fig. 18, the picture of the object corresponding to the foreground is moved so as to 
be displayed four pixels rightwards in a frame next to a directly previous reference 
frame. 
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In Fig. 18, the leftmost to fourth left pixels belong to the foreground area. In 
Fig. 1 8, fifth left to seventh left pixels in Fig. 1 8 belong to the mixed area which is the 
covered background area. In Fig. 1 8, the rightmost pixel belongs to the background 
area. 

Since the object corresponding to the foreground is moved to hide the object 
corresponding to the background, as time elapses, the components contained in the 
pixel values of the pixels belonging to the covered background area are switched from 
the background component picture to the foreground component picture at a certain 
time point of the time period corresponding to the shutter time. 

For example, the pixel value M, shown with a bold hne frame in Fig. 18, is 
represented by the equation (11): 

M = B02/v + B02/V + F07/v + F06/v 

•••(11). 

For example, the pixel value M, shown with a bold line frame in Fig. 18, 
contains the background component corresponding to one shutter time/v and the 
foreground component corresponding to three shutter time/v, the mixing ratio a of the 
fifth left pixel is 1/4. The sixth left pixel contains the background component 
corresponding to two shutter time/v and the foreground component corresponding to 
two shutter time/v, so the mixing ratio a is 1/2. The seventh left pixel contains the 
background component corresponding to three shutter time/v and the foreground 
component corresponding to one shutter time/v, so the mixing ratio a is 3/4. 
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Since the object corresponding to the foreground is a rigid body, such that the 
foreground picture is moved at an equal speed so as to be displayed four pixels 
towards right in the next frame, the first foreground component F07/v of the fourth left 
pixel in Fig. 18, with the first shutter time/v since the time of shutter opening, is equal 
to the second foreground component of the fifth left pixel in Fig. 1 8 corresponding to 
the second shutter time/v since the time of shutter opening. Siinilarly, the foreground 
component F07/v is equal to the foreground component of the sixth left pixel in Fig. 1 8 
corresponding to the third shutter time/v since the time of shutter opening and to the 
foreground component of the seventh left pixel in Fig. 1 8 corresponding to the fourth 
shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body, such that the 
foreground picture is moved at an equal speed so as to be displayed four pixels 
towards right in the next frame, the first foreground component F06/v of the third left 
pixel in Fig. 18, with the first shutter time/v since the time of shutter opening, is equal 
to the second foreground component of the fourth left pixel in Fig. 18 corresponding 
to the second shutter time/v since the time of shutter opening. Similarly, the 
foreground component F06/v is equal to the foreground component of the fifth left 
pixel in Fig. 18 corresponding to the third shutter time/v since the time of shutter 
opemng and to the foreground component of the sixth left pixel in Fig. 18 
corresponding to the fourth shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body, such that the 
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foreground picture is moved at an equal speed so as to be displayed four pixels 
towards right in the next frame, the first foreground component F05/v of the second 
left pixel in Fig. 18, with the first shutter time/v since the time of shutter opening, is 
equal to the third foreground component of the fourth left pixel in Fig. 18 
corresponding to the second shutter time/v since the time of shutter opening. 
Similarly, the foreground component F05/v is equal to the foreground component of 
the fourth left pixel in Fig. 18 corresponding to the third shutter time/v since the time 
of shutter opening and to the foreground component of the fifth left pixel in Fig. 18 
corresponding to the fourth shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body, such that the 
foreground picture is moved at an equal speed so as to be displayed four pixels 
towards right in the next frame, the first foreground component F04/v of the leftmost 
pixel in Fig. 1 8, with the first shutter time/v since the time of shutter opening, is equal 
to the second foreground component of the second left pixel in Fig. 18 corresponding 
to the second shutter time/v since the time of shutter opening. Similarly, the 
foreground component F04/v is equal to the foreground component of the third left 
pixel in Fig. 18 corresponding to the third shutter time/v since the time of shutter 
opening and to the foreground component of the fourth left pixel in Fig. 18 
corresponding to the fourth shutter time/v since the time of shutter opening. 

The foreground area corresponding to the moving object thus contains the 
motion blurring and hence may be said to be a distorted area. 
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Fig. 19 diagraiiiinatically shows pixel values of pixels on a line comprehending 
the uncovered background area in case the foreground is moving towards right in the 
drawing, with the pixels shown extended in the time axis direction. In Fig. 1 9, the 
movement quantity v of the foreground is 4. Since one frame is of short duration, it 
may be assumed that the object corresponding to the foreground is a rigid body and 
moving at an equal speed. In Fig. 19, the picture of the object corresponding to the 
foreground is moved so as to be displayed four pixels rightwards in a frame next to a 
directly previous frame. 

In Fig. 19, the leftmost to fourth left pixels belong to the background area. In 
Fig. 19, fifth left to seventh left pixels belong to the mixed area which is the covered 
background area. In Fig. 1 9, the rightmost pixel belongs to the background area. 

Since the object corresponding to the foreground which has hidden the object 
corresponding to the background is moved so as to be removed from a position ahead 
of the object corresponding to the background, as time elapses, the components 
contained in the pixel values of the pixels belonging to the covered background area 
are switched from the background component picture to the foreground component 
picture at a certain time point of the time period corresponding to the shutter time. 

For example, the pixel value M', shown with a bold line frame in Fig. 18, is 
represented by the equation (12): 

M' = F02/V + FO 1/v + B26/V + B26/v 

•••(12). 
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For example, the fifth left pixel contains the background component 
corresponding to three shutter time/v and the foreground component corresponding 
to one shutter time/v, the mixing ratio Od of the fifth left pixel is 3/4. The sixth left 
pixel contains the background component corresponding to two shutter time/v and the 
foreground component corresponding to two shutter time/v, so the mixing ratio CL is 
1/2. The seventh left pixel contains the background component corresponding to one 
shutter time/v and the foreground component corresponding to three shutter time/v, 
so the mixing ratio CL is 1/4. 
56. 

If the equations (1 1), (12) are generalized, the pixel value M is represented by 
the following equation (13): 

i -(13) 
where a is a mixing ratio, B is a pixel value of the background and Fi/v is the 
foreground component. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at an equal speed, with the movement quantity v being 4, the 
first foreground component FO 1/v of the fifth left pixel in Fig. 19, with the first shutter 
time/v since the time of shutter opening, is equal to the second foreground component 
of the sixth left pixel in Fig. 19 corresponding to the second shutter time/v since the 
time of shutter opening. Similarly, the FOl/v is equal to the foreground component of 
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the seventh left pixel in Fig. 1 9 corresponding to the third shutter time/v since the time 
of shutter opening and to the foreground component of the eighth left pixel in Fig. 19 
corresponding to the fourth shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at an equal speed, with the movement quantity v being 4, the 
first foreground component F02/v of the sixth left pixel in Fig. 19, with the first shutter 
time/v since the time of shutter opening, is equal to the second foreground component 
of the seventh left pixel in Fig. 1 9 corresponding to the second shutter time/v since the 
time of shutter opening. Similarly, the foreground component F02/v is equal to the 
foreground component of the eighth left pixel in Fig. 19 corresponding to the third 
shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at an equal speed, with the movement quantity v being 4, the 
first foreground component F03/v of the seventh left pixel in Fig. 19, with the first 
shutter time/v since the time of shutter opening, is equal to the second foreground 
component of the eighth left pixel in Fig. 19 corresponding to the second shutter time/v 
since the time of shutter opening. 

Although the number of times of the virtual splitting is four in the description 
with respect to Figs. 17 to 19, the number of times of the virtual splitting corresponds 
to the movement quantity v. The movement quantity v generally corresponds to the 
movement speed of the object corresponding to the foreground. For example, if the 
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object corresponding to the foreground is moving so as to be displayed four pixels 
rightwards in a frame next to a previous reference frame, the movement quantity v is 
4. The number of times of the virtual splitting is set to 4 in association with the 
movement quantity v. Similarly, if the object corresponding to the foreground is 
moving so as to be displayed six pixels rightwards in a frame next to a previous 
reference frame, the movement quantity v is 6, with the number of times of the virtual 
splitting being six. 

Figs. 20 and 2 1 show the relation between the foreground area, background area, 
and the mixed area, comprised of the covered background area and the uncovered 
background area, on one hand, and the foreground and background components 
corresponding to the split shutter time, on the other hand, as described above. 

Fig.20 shows an example of extraction of pixels of the foreground area, 
background area and the mixed area as extracted from a picture corresponding to an 
object moving before a still background. In the embodiment shown in Fig.20, an 
object corresponding to the foreground is moving horizontally with respect to the 
picture. 

The frame #n+l is a frame next to the frame #n, with the frame #n+2 being a 
frame next to the frame #n+l. 

Fig. 21 diagrammatically shows a model obtained on extracting pixels of the 
foreground area, background area and the mixed area, extracted in turn from one of 
the frames #n to #n+2, with the movement quantity v being 4, and on expanding the 
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pixel values of the extracted pixels along the time axis direction. 

Since the object corresponding to the foreground is moved, the pixel values of 
the foreground area are constituted by four different foreground components 
corresponding to the period of the shutter time/v. For example, the leftmost one of 
pixels of the foreground area shown in Fig. 2 1 are FO 1/v, F02/v, F03/v and F04/v. That 
is the pixels of the foreground area contain are corrupted with motion blurring. 

Since the object corresponding to the background is at a standstill, the light 
corresponding to the background input to the sensor 1 1 during the time corresponding 
to the shutter time is not changed. In this case, the pixel values of the background are 
free of the motion blurring. 

The pixel values or the pixels belonging to the mixed area composed of the 
covered background area or the uncovered background area are comprised of the 
foreground and background components. 

A model comprised of neighboring pixels in a row in plural frames, in which the 
pixel values of pixels lying at the same position on a frame are developed in the time 
axis direction, with the picture corresponding to an object being moved, is explained. 
For example, if the picture corresponding to the object is moving horizontally with 
respect to the picture, the pixels arrayed on the same row on the picture may be 
selected as the pixels in a row in a picture. 

Fig.22 diagrammatically shows a model obtained on temporally expanding the 
pixel values of pixels arrayed in a row of each of three frames of a picture of a 
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photographed object corresponding to a still background, with the developed pixels 
being at the same positions on the respective frames. The frame #n is the frame next 
to the frame #n-l, with the frame #n+l being the frame next to the frame #n. The 
remaining frames are ternied in similar manner. 

The pixel values of BOl to B12 shown in Fig.22 are those of pixels 
corresponding to the object of the still background. Since the objet corresponding to 
the background is at a standstill, the pixel values of the corresponding pixels in the 
frames #n- 1 ti frame n+1 are not changed. For example, the pixel in the frame #n and 
the pixel in the frame #n+l, corresponding to the positions of the pixels having pixel 
values of B05 in the frame #n- 1, are of pixel values of BOS. 

Fig.23 shows pixel values of neighboring pixels in a row in each of three frames 
of a photographed picture of an object corresponding to the foreground moving 
rightwards in Fig.23, along with the object corresponding to the still background, with 
the pixel values being shown developed along the time axis direction. The model 
shown in Fig.23 includes a covered background area. 

In Fig.23, the object corresponding to the foreground is a rigid body and may 
be assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame. So, the movement quantity v of the foreground is 4, with the number of times 
of the virtual splitting being 4. 

For example, the foreground component of the leftmost pixel of the frame #n- 1 
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in Fig.23 , with the first shutter time /v since the opening of the shutter, is F 1 2/v, whilst 
the foreground component of the second left pixel, with the second shutter time /v 
since the opening of the shutter, is also F12v. The foreground component of the third 
left pixel in Fig.23, with the third shutter time/v since the opening of the shutter, and 
the foreground component of the fourth left pixel in Fig.23, with the fourth shutter 
time/v since the opening of the shutter, are each F12/v. 

For example, the foreground component of the leftmost pixel of the fi-ame #n' 1 
in Fig.23, with the second shutter time /v since the opening of the shutter, is Fl 1/v, 
whilst the foreground component of the second left pixel, with the third shutter time 
/v since the opening of the shutter, is also Fl Iv. The foreground component of the 
third left pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, 
isFU/v. 

The foreground component of the leftmost pixel of the frame #n- 1 in Fig.23, 
with the third shutter time /v since the opening of the shutter, is FlO/v, whilst the 
foreground component of the second left pixel, with the fourth shutter time /v since 
the opening of the shutter, is also FlOv. The foreground component of the leftmost 
pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, is F09/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the second left pixel of the fi-ame #n' 1 in Fig.23, with the 
first shutter time/v as from the shutter opening time, is BO 1/v. The background 
component of the third left pixel of the frame #n* 1 in Fig.23, with the first and second 
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shutter time/v as from the shutter opening time, is B02/v, while the background 
component of the fourth left pixel of the frame #n- 1 in Fig.23, with the first to third 
shutter time/v as from the shutter opening time, is B03/v. 

In the frame #n- 1 in Fig.23, the leftmost pixel belongs to the foreground area, ' 
while the second to fourth left pixels belong to the mixed area which is the covered 
background area. 

The fifth to twelfth left pixels of the frame #n- 1 in Fig.23 belong to the 
background area, with the corresponding pixel values being B04 to B 11, respectively. 

The first to fifth pixels of the frame #n- 1 in Fig.23 belong to the background 
area. The foreground component in the foreground area of the frame #n, with the 
shutter time/v, is one of F05v to F12/v. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the fifth left pixel of the frame #n-in Fig.23, with 
the fu-st shutter time /v since the opening of the shutter, is F 1 2/v, whilst the foreground 
component of the sixth left pixel, with the second shutter time /v since the opening of 
the shutter, is also F12v. The foreground component of the seventh left pixel in 
Fig.23, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the eighth left pixel in Fig.23, with the fourth shutter time/v 
since the opening of the shutter, are each F12/v. 



m # 

57 

The foreground component of the fifth left pixel of the fi-ame #n in Fig.23, with 
the second shutter time /v since the opening of the shutter, is Fll/v, whilst the 
foreground component of the sixth left pixel, with the third shutter time /v since the 
opening of the shutter, is also Fl Iv. The foreground component of the seventh left 
pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, is F 1 1/v. 

The foreground component of the fifth left pixel of the fi-ame #n in Fig.23, with 
the third shutter time /v since the opening of the shutter, is FIO/v, whilst the 
foreground component of the sixth left pixel, with the fourth shutter time /v since the 
opening of the shutter, is also FlOv. The foreground component of the fifth left pixel 
m Fig.23, with the fourth shutter time/v since the opening of the shutter, is F09/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the sixth left pixel of the fi-ame #n in Fig.23, with the first 
shutter time/v as from the shutter opening time, is B05/v. The background component 
of the seventh left pixel of the frame #n in Fig.23, with the first and second shutter 
time/v as from the shutter opening time, is B06/v, while the background component 
of the eighth left pixel of the frame #n in Fig.23, with the first to third shutter time/v 
as from the shutter opening time, is B07/v. 

In the frame #n- 1 in Fig.23, the first to ninth left pixels belong to the foreground 
area, while the sixth to eighth left pixels belong to the mixed area which is the covered 
background area. 

The first to ninth to twelfth left pixels of the frame #n+l in Fig.23 belong to the 
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foreground area, with the pixel values being BOS to Bl 1, respectively. 

The first to ninth pixels of the frame #n+l in Fig.23 belong to the foreground 
area. The foreground component in the foreground area of the fi-ame #n+l, with the 
shutter time/v, is one of FOlv to F12/v. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the ninth left pixel of the fi-ame #n+l in Fig.23, 
with the first shutter time /v since the opening of the shutter, is F12/v, whilst the 
foreground component of the tenth left pixel, with the second shutter time /v since the 
opening of the shutter, is also F12v. The foreground component of the eleventh left 
pixel in Fig.23, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the twelfth left pixel in Fig.23, with the fourth shutter time/v 
since the opening of the shutter, are each F12/v. 

The foreground component of the ninth left pixel of the frame #n+l in Fig.23, 
with the second shutter time /v since the opening of the shutter, is Fl 1/v, whilst the 
foreground component of the tenth left pixel, with the third shutter time /v since the 
opening of the shutter, is also Fl Iv. The foreground component of the eleventh left 
pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, is Fl I/v. 

The foreground component of the ninth left pixel of the frame #n+l in Fig.23, 
with the third shutter time /v since the opening of the shutter, is FlO/v, whilst the 
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foreground component of the tenth left pixel, with the fourth shutter time /v since the 
opening of the shutter, is also FlOv. The foreground component of the ninth left pixel 
of the fi-ame #n+lin Fig.23, with the fourth shutter time/v since the opening of the 
shutter, is F09/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the tenth left pixel of the frame #n+l in Fig.23, with the 
first shutter time/v as from the shutter opening time, is B09/v. The background 
component of the eleventh left pixel of the frame #n+l in Fig.23, with the first and 
second shutter time/v as from the shutter opening time, is B 1 0/v, while the background 
component of the twelfth left pixel of the frame #n+l in Fig.23, with the first to third 
shutter time/v as from the shutter opening time, is Bl 1/v. 

In the frame #n+l in Fig.23, the tenth to twelfth left pixels correspond to the 
mixed area which is the covered background area. 

Fig.24 diagrammatically shows a picture obtained on extracting the foreground 
component from the pixel values shown in Fig.23. 

Fig. 2 5 shows neighboring pixels in a row of each of three frames of a 
photographed picture of the foreground corresponding to an object moving rightwards 
in the drawing, along with the still background. In Fig.25, there is also shown the 
uncovered background area. 

In Fig.25, the object corresponding to the foreground is a rigid body and may 
be assumed to be moving at a constant speed, with the foreground picture being moved 
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so that the foreground picture will be displayed four pixels rightwards in the next 
frame. So, the movement quantity v of the foreground is 4. 

For example, the foreground component of the leftmost pixel of the frame #n' 1 
in Fig.25, with the first shutter time /v since the opening of the shutter, is F13/v, whilst 
the foreground component of the second left pixel, with the second shutter time /v 
since the opening of the shutter, is also F13v. The foreground component of the third 
left pixel in Fig.23 , with the second shutter time/v since the opening of the shutter, and 
the foreground component of the fourth left pixel in Fig.25, with the fourth shutter 
time/v since the opening of the shutter, are each F13/v. 

For example, the foreground component of the second left pixel of the frame 
#n' 1 in Fig.23, with the first shutter time /v since the opening of the shutter, is F14/v, 
whilst the foreground component of the third left pixel, with the second shutter time 
/v since the opening of the shutter, is also F14v. The foreground component of the 
third left pixel in Fig.25, with the first shutter time/v since the opening of the shutter, 
isF15/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the leftmost pixel of the frame #n-l in Fig.25, with the 
second to fourth shutter time/v as from the shutter opening time, is BOl/v. The 
background component of the second left pixel of the frame #n-l in Fig.25, with the 
third and fourth shutter time/v as from the shutter opening time, is B26/v, while the 
background component of the third left pixel of the frame #n-l in Fig.25, with the 
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fourth shutter time/v as from the shutter opening time, is B27/v. 

In the frame #n' 1 in Fig.25, the first to third left pixel belongs to the mixed area 
which is the covered background area. 

The fourth to twelfth left pixels of the frame #n'l in Fig.25 belong to the ' 
foreground area, with the foreground component of the foreground of the frame being 
one of F13v to F24v. 

The first to fourth left pixels of the frame #n in Fig.25 belong to the background 
area, with the pixel values being B25 to B28, respectively. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the fifth left pixel of the frame #n in Fig.23, with 
the first shutter time /v since the opening of the shutter, is F 13/v, whilst the foreground 
component of the sixth left pixel, with the second shutter time /v since the opening of 
the shutter, is also F13v. The foreground component of the seventh left pixel in 
Fig.25, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the eighth left pixel in Fig.25, with the fourth shutter time/v 
since the opening of the shutter, are each F13/v. 

The foreground component of the sixth left pixel of the frame #n in Fig.23, with 
the first shutter time /v since the opening of the shutter, is F 14/v, whilst the foreground 
component of the seventh left pixel, with the second shutter time /v since the opening 
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blurring contained in the temperature data generated by the movement of the object 
being measured to calculate the more accurate temperature difference of respective 
object portions. 

Fig.73 shows an illustrative structure of a signal processing apparatus according 
to the present invention for weighing measurement. The pressure area sensor 501 is 
made up of plural pressure sensors to measure the load per unit planar area, that is the 
pressure. . The pressure area sensor 501 is of a structure comprised of a two- 
dimensional array on a floor surface of plural pressure sensors 51 1-1-1 to 51 1-M-N. 
When an object 5 12, the weight of which is being measured, is moved on the pressure 
area sensor 501, the pressure area sensor 501 measures the pressure applied to each 
of the pressure sensors 511-1-1 to 511-M-N to generate weight data for each of 
measured ranges of the pressure sensors 5 11- 1 -1 to 5 1 1-M-N to output the generated 
weight data to the signal processor 502. 

The pressure sensors 511-1-1 to 511-M-N are each made up of a sensor 
exploiting double refraction produced when an external force is applied to a 
transparent elastic material, or the so-called photoelasticity. 

The pressure area sensor 501 in its entirety may be constructed by a sensor 
exploiting the photoelasticity. 

Fig.75 illustrates the load associated with the weight of each part of the object 
512 apphed to respective ones of the pressure sensors 511-1-1 to 511 -M-N making up 
the pressure area sensor 501. 
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foreground component of the tenth left pixel, with the second shutter time /v since the 
opening of the shutter, is also F13v. The foreground component of the eleventh left 
pixel in Fig.25, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the twelfth left pixel in Fig.25, with the fourth shutter time/v 
since the opening of the shutter, are each F13/v. 

The foreground component of the tenth left pixel of the fi-ame #n+l in Fig.25, 
with the first shutter time /v since the opening of the shutter, is F14/v, whilst the 
foreground component of the eleventh left pixel, with the second shutter time /v since 
the opening of the shutter, is also F14v. The foreground component of the twelfth left 
pixel in Fig.25, with the first shutter time/v since the opening of the shutter, is FI5/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the ninth left pixel of the frame #n+l in Fig.25, with the 
second to fourth shutter time/v as from the shutter opening time, is B33/v. The 
background component of the tenth left pixel of the frame #n+l in Fig.25, with the 
third and fourth shutter time/v as from the shutter opening time, is B34/v, while the 
background component of the eleventh left pixel of the frame #n+l in Fig.25, with the 
fourth shutter time/v as from the shutter opening time, is B35/v. 

In the frame #n+l in Fig.25, the ninth to eleventh left pixels correspond to the 
mixed area which is the covered background area. 

In Fig.25, the twelfth left pixel of the frame #n+l belong to the foreground area. 
The foreground component with the shutter time/v in the foreground area of frame 
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#n+l is one of F13vto F16v. 

Fig. 2 6 diagrainiTiatically shows a picture obtained on extracting the foreground 
component from the pixel values shown in Fig.25. 

Reverting to Fig. 10, the area specifying unit 103 associates a flag, indicating ' 
that a given picture belong to the foreground area, a background area, a covered 
background area or an uncovered background area, from pixel to pixel, using pixel 
value of plural frames, and routes the resulting areal information to the mixing ratio 
calculating unit 104 and to the motion blurring adjustment unit 106. 

Based on the pixel values of plural frames and the areal infonnation, the mixing 
ratio calculating unit 104 computes the mixing ratio CL for each of the pixels contained 
in the mixed area, and sends the computed mixing ratio a to the 
foreground^ackground separating unit 105. 

Based on the pixel values of the plural frames, areal infonnation and the mixing 
ratio OC, the foreground/background separating unit 105 extracts the foreground 
component picture made up only of the foreground component to send the extracted 
component picture to the motion blurring adjustment unit 106. 

Based on the foreground component picture sent from the 
foreground/background separating unit 105, the motion vector sent from the motion 
detection unit 1 02 and on the areal infonnation sent from the area specifying unit 1 03, 
the motion blurring adjustment unit 106 adjusts the quantity of the motion blurring 
contained in the foreground component picture to output the foreground component 
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picture adjusted for the motion blurring. 

Referring to the flowchart of Fig.27, the processing for adjusting the motion 
blurring caused by the signal processor 12 is explained. At step SI 01, the area 
specifying unit 103 executes the area specifying processing for generating the areal 
infomiation indicating to which of the foreground area, background area, covered 
background area or the uncovered background area belong the pixels of the input 
picture, from one pixel of the input picture to another. The area specifying processing 
will be explained subsequently by referring to the flowchart of Fig.36. The area 
specifying unit 103 sends the generated area information to the mixing ratio 
calculating unit 104. 

Meanwhile, the area specifying unit 103 at step SI 01 may generate the areal 
information indicating to which of the foreground area, background area or the mixed 
area belong the pixels of the input picture, from one pixel of the input picture to 
another, based on the input picture. In this case, no distinction is made between the 
covered background area and the uncovered background area. In this case, the 
foreground/background separating unit 105 and the motion blurring adjustment unit 
106 decide whether the mixed area is the covered background area or the uncovered 
background area, based on the direction of the motion vector. For example, if the 
foreground area, mixed area and the background area are arrayed sequentially in 
association with the direction of the motion vector, the mixed area is verified to be the 
covered background area, whereas, if the background area, mixed area and the 
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foreground area are arrayed sequentially in association with the direction of the motion 
vector, the mixed area is verified to be the uncovered background area. 

At step S102, the mixing ratio calculating unit 104 calculates the mixing ratio 
a, fi-om one pixel contained in the mixing area to another, based on the input picture 
and the area infonnation. The processing for computing the mixing ratio will be 
explained in detail subsequently by referring to the flowchart of Fig.46. The mixing 
ratio calculating unit 104 sends the computed mixing ratio a to the 
foreground/background separating unit 105. 

At step S103, the foreground/background separating unit 105 extracts the 
foreground component fi-om the input picture, based on the motion vector and the areal 
information, to send the extracted component to the motion blurring adjustment unit 
106 as the foreground component picture. 

At step S 104, the motion blurring adjustment unit 106 generates a processing 
unit for indicating a position on the picture of pixels arrayed consecutively in the 
movement direction of each of the uncovered background area, foreground area and 
the covered background area, based on the motion vector and on the area information, 
to adjust the quantity of the motion blurring contained in the foreground component 
corresponding to the processing unit. The processing for adjusting the quality of the 
motion blurring will be explained subsequently by referring to the flowchart of Fig. 63 . 

At step S 1 05, the signal processor 1 2 verifies whether or not the processing has 
been finished for the entire picture. If die signal processor 12 has verified that the 
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processing has not been finished fort the entire picture, it proceeds to step S104 to 
repeat the processing for adjusting the quantity of the motion blurring for the 
foreground component corresponding to the processing unit. 

If, at step S 1 06, it is verified that the processing has been finished for the entire 
picture, the processing is teraiinated. 

In this manner, the signal processor 12 is able to separate the foreground and 
the background from each other to adjust the quantity of the motion blurring contained 
in the foreground. That is, the signal processor 12 is able to adjust the amount of 
motion blurring contained in sample data as pixel value of the foreground pixel. 

In the following, illustrative structures of the area specifying unit 103, mixing 
ratio calculating unit 104, foreground/background separating unit 105 and the motion 
blurring adjustment unit 106 are hereinafter explained. 

Fig. 2 8 is a block diagram showing an illustrative structure of the area specifying 
unit 103. A frame memory 121 stores an input picture on the frame basis. When a 
frame being processed is a frame #n, the frame memory 121 stores a frame #n'2, as 
a frame two fraines before the frame #n, a frame #n-l, as a frame one frame before 
the frame #n, a frame #n+I, as a frame one frame after the frame #n, and a frame 
#n+2, as a frame two frames after the frame #n. 

A still/movement discriminating unit 122-1 reads out a pixel value of a pixel 
of the frame #n+2 lying at the same position as the position on the picture of the pixel 
of the frame #n being area-specified, and a pixel value of a pixel of the frame #n+l 
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lying at the same position as the position on the picture of the pixel of the frame #n 
being area-specified, from the frame memory 12 1 , to calculate an absolute value of the 
difference of the read-our pixel values. The still/movement discriminating unit 122- 1 
verifies whether or not the absolute value of the difference between the pixel value of 
the fraine #n+2 and the frame #n+l is larger than a predetermined threshold value Th. 
If it is verified that the absolute value of the difference is larger than the threshold 
value Th, the still/movement discriminating unit 122-1 routes a still/movement 
decision specifying the movement decision to an area decision unit 123-1. If it is 
verified that the absolute value o the difference between the pixel value of the frame 
#n+2 nd the pixel value of the frame #n+l is not larger than the threshold value Th, 
the still/movement discriminating unit 122-1 routes a still/movement decision 
specifying the still decision to an area decision unit 123-1 . 

A still/movement discriminating unit 122-2 reads out a pixel value of a pixel of 
the frame #n+l lying at the same position as the position on the picture of the pixel of 
the frame #n being area-specified, and a pixel value of a pixel of the frame #n+l lying 
at the same position as the position on the picture of the pixel of the frame #n being 
area-specified, from the frame memory 121, to calculate an absolute value of the 
difference of the read-our pixel values. The still/movement discriminating unit 122-2 
verifies whether or not the absolute value of the difference between the pixel value of 
the frame #n+l and the frame #n is larger than a predetermined threshold value Th. 
If it is verified that the absolute value of the difference between the is larger than the 



69 

threshold value Th, the still/movement discriminating unit 122-1 routes a 
still/movement decision specifying the movement decision to an area decision unit 
123-land to an area decision unit 123-2. If it is verified that the absolute value of the 
difference between the pixel value of the pixel of the frame #n+l and that of the pixel 
of the frame #n is not larger than the threshold value Th, the still/movement 
discriininating unit 122- 1 routes a still/movement decision specifying the still decision 
to an area decision unit 123-1 and to an area decision unit 123-2. 

A still/movement discriminating unit 122-3 reads out a pixel value of a pixel of 
the frame i^n lying at the same position as the position on the picture of the pixel of the 
frame #n being area-specified, and a pixel value of a pixel of the frame #n' 1 lying at 
the same position as the position on the picture of the pixel of the frame #n being area- 
specified, from the frame memory 12 1 , to calculate an absolute value of the difference 
of the read-our pixel values. The still/movement discriininating unit 122-3 verifies 
whether or not the absolute value of the difference between the pixel value of the 
frame #n and the frame #n- 1 is larger than a predetermined threshold value Th. If it 
is verified that the absolute value of the difference between the pixel values is larger 
than the threshold value Th, the still/movement discriminating unit 122-3 routes a 
still/movement decision specifying the movement decision to an area decision unit 
123-land to an area decision unit 123-3. If it is verified that the absolute value of the 
difference between the pixel value of the pixel of the frame #n and that of the pixel of 
the frame #n-l is not larger than the threshold value Th, the still/movement 
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discriminating unit 122-3 routes a still/movement decision specifying the still decision 
to an area decision unit 123-2 and to an area decision unit 123-3. 

A still/movement discriminating unit 1 22-4 reads out the pixel value of the pixel 
of the frame #n* 1 lying at the same position as the position on the picture of the pixel 
of the frame #'n being area-specified, and the pixel value of the pixel of the frame #n'2 
lying at the same position on the picture of the pixel of the frame #n being area- 
specified, to calculate the absolute value of the difference of the pixel values. The 
still/movement discriminating unit 122-4 verifies whether or not the absolute value of 
the difference of the pixel value of the frame #n-l and the pixel value of the frame 
#n-2 is larger than the predetennined threshold value Th. If the absolute value of the 
difference between the pixel value of the frame #n' 1 and the pixel value of the frame 
#n-2 is verified to be larger than the threshold value Th, a still/movement decision 
indicating the decision for movement is routed to the area decision unit 123-3. If it is 
verified that the absolute value of the difference between the pixel value of the frame 
#n' 1 and the pixel value of the frame #n-2 is not larger than the threshold value Th, the 
still/movement discriminating unit 122-4 routes a still/movement decision indicating 
the still decision to the area decision unit 123-3. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-1 indicates still and the still/movement decision routed from the 
still/movement discriminating unit 122-2 indicates movement, the area decision unit 
123-1 decides that the pixel on the frame #n being area-specified belongs to the 
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uncovered background area and sets "1" in an uncovered background area decision 
flag associated with the pixel being area-specified for indicating that the pixel belongs 
to the uncovered background area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-1 indicates movement and the still/movement decision routed from the 
still/movement discriminating unit 122-2 indicates still, the area decision unit 123-1 
decides that the pixel on the frame #n being area-specified does not belong to the 
uncovered background area and sets "0" in an uncovered background area decision 
flag associated with the pixel being area-specified for indicating that the pixel does not 
belong to the uncovered background area. 

The area decision unit 123-1 routes the uncovered background area decision 
flag, having "1" or "0" set in this maimer, to a decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates still and the still/movement decision routed from the 
still/movement discriminating unit 122-3 indicates still, the area decision unit 123-2 
decides that the pixel on the frame #n being area-specified belongs to the still area and 
sets "1" in a still area decision flag associated with the pixel being area-specified for 
indicating that the pixel belongs to the uncovered background area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates movement or the still/movement decision routed from the 
still/movement discriminating unit 122-3 indicates movement, the area decision unit 
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123-2 decides that the pixel on the frame #n being area-specified does not belong to 
the still area and sets "0" in a still area decision flag associated with the pixel being 
area-specified for indicating that the pixel does not belong to the still area. 

The area decision unit 123-2 routes the still area decision flag, thus having "1" ' 
or "0" set therein, to the decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates movement and the still/movement decision routed from the 
still/movement discriminating unit 122-3 indicates movement, the area decision unit 
123-2 decides that the pixel on the frame #n being area-specified belongs to the 
movement area and sets " 1" in a movement area decision flag associated with the pixel 
being area- specified for indicating that the pixel belongs to the movement area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates still or the still/movement decision routed from the still/movement 
discriminating unit 122-3 indicates still, the area decision unit 123-2 decides that the 
pixel on the frame #n being area-specified does not belong to the movement area and 
sets "0" in a movement area decision flag associated with the pixel being area- 
specified for indicating that the pixel does not belong to the movement area. 

The area decision unit 123-2 routes the movement area decision flag, thus 
having "1" or "0" set therein, to the decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-3 indicates movement and the still/movement decision routed from the 
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still/movement discriminating unit 122-4 indicates still, the area decision unit 123-3 
decides that the pixel on the frame #n being area-specified belongs to the uncovered 
background area and sets "1" in a covered background area decision flag associated 
v^ith the pixel being area-specified for indicating that the pixel belongs to the covered 
background area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-3 indicates still or the still/movement decision routed from the still/movement 
discriminating unit 122-4 indicates movement, the area decision unit 123-3 decides 
that the pixel on the frame #n being area-specified does not belong to the covered 
background area and sets "0" in a covered background area decision flag associated 
with the pixel being area-specified for indicating that the pixel does not belong to the 
covered background area. 

The area decision unit 123-3 routes the covered background area decision flag, 
thus having " 1 " or "0" set therein, to the covered background area decision flag storage 
memory 124. 

The decision flag storage memory 124 stores the uncovered background area 
decision flag, sent from the area decision unit 123-1, the still area decision flag, sent 
from the area decision unit 123-2, the movement area decision flag, sent from the area 
decision unit 123-2, and the uncovered background area decision flag, sent from the 
area decision unit 123-3. 

The decision flag storage memory 124 sends the uncovered background area 
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decision flag, still area decision flag, movement area decision flag and the covered 
background area decision flag to a synthesis unit 125. Based on the uncovered 
background area decision flag, still area decision flag, movement area decision flag 
and the covered background area decision flag, supplied from the decision flag storage 
memory 124, the synthesis unit generates the area infonnation indicating to which of 
the uncovered background area, still area, movement area and the covered background 
area belong the respective pixels, and routes the infonnation so generated to a decision 
flag storage frame memory 126. 

The decision flag storage frame memory 126 stores the area infonnation, 
supplied from the synthesis unit 125, while outputting the area infrynnation stored 
therein. 

Referring to Figs.29 to 33, a typical processing by the area specifying unit 103 
is explained. 

When an object corresponding to the foreground is moving, the position of the 
picture corresponding to the object on the picture screen is changed from frame to 
fraine. Referring to Fig.29, a picture corresponding to an object at a position Yn(x, y) 
in a frame #n is positioned at Yn+l(x,y) at the next frame #n+l. 

Fig.30 diagrammatically shows a model of pixel values of a row of pixels 
neighboring to one another along the moving direction of the picture corresponding 
to the foreground. For example, if the movement direction of the picture 
corresponding to the foregroimd is horizontal relative to the picture screen, the 
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diagrairanatic view of Fig.30 shows a model in which pixel values of pixels 
neighboring to one another on one line are developed in the time axis direction. 

In Fig.30, the line in the frame #n is the same as one in the frame #n+l. 

The components of the foreground corresponding to the object contained in the 
second to the thirteenth pixels as counted from left in the frame #n are included in the 
sixth to seventeenth pixels as counted from the left of the frame #n+l. 

The pixels belonging to the covered background area in the frame #n are the 
eleventh to thirteenth pixels as counted from left, whilst the pixels belonging to the 
uncovered background area are the second to fourth pixels as counted from left. The 
pixels belonging to the covered background area in the frame #n+l are the fifteenth 
to seventeenth pixels as counted from left, whilst the pixels belonging to the uncovered 
background area are the sixth to eighth pixels as counted from left. 

In the example shown in Fig.30, since the foreground component in the frame 
#n are moved by four pixels in the frame #n+l, the movement quantity v is 4. The 
number of times of the virtual splitting corresponds to the movement quantity and is 
equal to 4. 

The change in the pixel values of pixels belonging to the mixed area ahead and 
at back of the frame being considered is explained. 

In the frame #n shown in Fig.31, in which the background is still and the 
movement quantity of the foreground v is 4, pixels belonging to the covered 
background area are fifteenth to seventeenth pixels from left. Since the movement 
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quantity v is 4, the fifteenth to seventeenth pixels from left in the directly previous 
fi"ame #n- 1 contain only the background components and belong to the background. 
The fifteenth to seventeenth pixels from left in the fiuther previous frame #n'2 contain 
only the background components and belong to the background area. 

Since the object corresponding to the background is still, the pixel value of the 
fifteenth pixel from the left of the frame #n' 1 is not changed from the pixel value of 
the fifteenth pixel from the left of the fraine #n-2. Similarly, the pixel value of the 
sixteenth pixel from the left of the frame #n' 1 is not changed from the pixel value of 
the sixteenth pixel from the left of the frame #n'2, whilst the pixel value of the 
seventeenth pixel from the left of the frame #n' 1 is not changed from the pixel value 
of the seventeenth pixel from the left of the frame #n-2. 

That is, the pixels of the frame #n*l and the frame #n-2 corresponding to the 
pixels belonging to the covered background area in the frame #n are comprised only 
of the background components and are not changed, so that the absolute value of the 
difference is substantially 0. So, the still/movement decision on the pixels of the frame 
#n' 1 and frame #n'2 corresponding to the mixed area in the frame #n is made as being 
still by the still/moving discriminating unit 122-4. 

Since the pixels belonging to the covered background area in the frame #n 
contain the foreground components, the corresponding pixel values differ from those 
in which the pixels are comprised only of background components in the fraine #n- 1 . 
Therefore, the pixels belonging to the mixed area in the frame #n and the 
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corresponding pixels of the frame #rv 1 are verified to be moving pixels by the 
still/moving discriminating unit 122-3. 

When fed with the result of still/movement decision indicating the movement 
from the still/moving discriminating unit 122-3 and with the result of still/movement 
decision indicating the still from the still/moving discriminating unit 122-4, the area 
decision unit 123-3 decides that the pixel in question belongs to the covered 
background area. 

The pixels contained in the uncovered background area in the frame #n in which 
the background is still and the movement quantity v of the foreground is 4 are second 
to fourth pixels as counted from left. Since the movement quantity v is 4, the second 
to fourth pixels from left in the next frame #n+l contain only the background 
components and belong to the background area. In the second next frame #n+2, the 
second to fourth pixels from left contain only the background components and belong 
to the background area. 

Since the object corresponding to the background is still, the pixel value of the 
second pixel from left of the frame #n+2 is not changed from the pixel value of the 
second pixel from left of the frame #n+l . Similarly, the pixel value of the second pixel 
from left of the frame #n+2 is not changed from the pixel value of the second pixel 
from left of the frame #n+l, whilst the pixel value of the third pixel from left of the 
frame #n+2 is not changed from the pixel value of the fourth pixel from left of the 
frame #n+l. 
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That is, the pixels of the frame #n+ 1 and frame #n+2 corresponding to the 
pixels belonging to the uncovered background area in the frame #n are composed only 
of background components and are not changed in the pixel values. So, the absolute 
value of the difference is approximately zero. Therefore, the pixels of the frame #n+l 
and frame #n+2 corresponding to the pixels belonging to the mixed area in the frame 
#n are decided by the still/moving discriminating unit 122-1 to be still pixels. 

The pixels belonging to the uncovered background area in the frame #n contain 
the foreground components and hence differ in pixel values from the pixels in the 
frame #n+l composed only of the background components. So, the pixels belonging 
to the mixed area in the frame #n and those of the corresponding frame #n* 1 are 
decided by the still/moving discriminating unit 122-2 to be moving pixels. 

The area decision unit 123-1 is fed in this manner with the result indicating 
movement from the still/moving discriminating unit 122-2. If fed with the result 
indicating still from the still/moving discriminating unit 122-1, the area decision unit 
123- 1 decides that the corresponding pixel belongs to the uncovered background area. 

Fig.33 shows decision conditions of the area specifying unit 103 in the frame 
#n. When the pixel of the frame #n-2 at the same position as the position on the 
picture of the pixel of the frame #n being verified and the pixel of the frame #n- 1 at 
the same position as the position on the picture of the pixel of the frame #n being 
verified, are decided to be still, whilst the pixel of the frame #n* 1 at the same position 
as the position on the picture of the pixel of the frame #n being verified and the pixel 
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of the frame #n are decided to be moving, the area specifying imit 103 decides that the 
pixel of the frame #n being verified belongs to the covered background area. 

When the pixel of the frame #n* 1 at the same position as the position on the 
picture of the pixel of the frame #n being verified and the pixel of the frame #n are 
decided to be still, whilst the pixel of the frame #n and the pixel of the frame #n+l at 
the same position as the position on the picture of the pixel of the frame #n being 
verified are decided to be still, the area specifying unit 1 03 decides that the pixel of the 
frame #n being verified belongs to the still area. 

When the pixel of the frame #n' 1 at the same position as the position on the 
picture of the pixel of the frame #n being verified and the pixel of the frame #n are 
decided to be moving, whilst the pixel of the frame #n and the pixel of the frame #n+ 1 
at the same position as the position on the picture of the pixel of the frame #n being 
verified are decided to be still, the area specifying unit 103 decides that the pixel of the 
frame #n being verified belongs to the moving area. 

When the pixel of the frame #n and the pixel of the frame #n+l at the same 
position as the position on the picture of the pixel of the frame #n being verified are 
decided to be moving and when the pixel of the frame #n+l at the same position as the 
position on the picture of the pixel of the frame #n being verified and the pixel of the 
frame #n+l at the same position as the position on the picture of the pixel of the frame 
#n being verified and the pixel of the frame #n+2 at the same position as the position 
on the picture of the pixel of the frame #n being verified are decided to be still, the 
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area specifying unit 103 decides that the pixel of the frame #n being verified belongs 
to the uncovered background area. 

Fig.34 shows an example of the area decision by the area specifying unit 103. 
In Fig. 34 A, a pixel decided to belong to the covered background area is shown in 
white. In Fig.34B, a pixel decided to belong to the uncovered background area is 
shown in white. 

In Fig.34C, a pixel decided to belong to the moving area is shown in white. In 
Fig.34D, a pixel decided to belong to the still area is shown in white. 

Fig. 3 5 shows the area information representing the mixed area, among the area 
information output by the decision flag storage frame memory 126, as picture. In 
Fig.35, the pixel decided to belong to the covered background area or the uncovered 
background area, that is to the mixed area, is shown in white. The area information 
indicating the mixed area, output by the decision flag storage frame memory 126, 
indicates a textured portion surrounded by an untextured portion in the foreground 
area and the mixed area. 

Referring to the flowchart of Fig.36, the processing for area identification by 
the area specifying unit 103 is explained. At step S121, the frame memory 121 
acquires pictures of the frame #n-2 to frame #n+2, inclusive the frame #n. 

At step S122, the still/moving discriminating unit 122-3 checks whether or not 
the pixels at the same position of the frame #n' 1 and the frame #n are still. If the 
pixels are decided to be still, the program moves to step SI 23 where the still/moving 
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discriminating unit 122-2 checks whether or not the pixels at the same position of the 
frame #n and the frame #n+l are still. 

If, at step SI 23, the pixels at the same position of the frame #n and the pixel of 
the frame #n+l are decided to be still, the program moves to step S 124 where the area 
decision unit 123-2 sets "1" in the still area decision flag corresponding to the pixel of 
the area being verified for indicating that the pixel belongs to the still area. The area 
decision unit 123-2 sends the still area decision flag to the decision flag storage 
memory 124. The program then moves to step S125. 

If at step SI 22 the pixels at the same position of the frame #n-l and the frame 
#n are decided to be moving or if at step SI 23 the pixels at the same position of the 
frame #n and the frame #n+l are decided to be moving, the pixel of the frame #n does 
not belong to the still area, so the processing at stepS124 is skipped and the program 
moves to step S125. 

At step S 125, the still/moving discriminating unit 122-3 checks whether ornot 
the pixels at the same position of the frame #n-l and the frame #n are moving. If the 
pixels are decided to be moving, the program moves to step SI 26 where the 
still/moving discriminating unit 122-2 decides whether or not the pixels at the same 
position of the frame #n and the frame #n+l are moving. 

If, at step S 1 26, the pixels at the same position of the frame #n and the pixel of 
the frame #n+l are decided to be moving, the program moves to step S 127 where the 
area decision unit 123-2 sets "1" in the movung area decision flag corresponding to the 
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pixel of the area being verified for indicating that the pixel belongs to the moving area. 
The area decision unit 123-2 sends the moving area decision flag to the decision flag 
storage memory 124. The program then moves to step S128. 

If at step S125 the pixels at the same position of the fi-ame #n* 1 and the frame 
#n are decided to be still or if at step SI 26 the pixels at the same position of the frame 
#n and the frame #n+l are decided to be still, the pixel of the frame #n does not belong 
to the moving area, so the processing at stepS127 is skipped and the program moves 
to step S128. 

At step S 128, the still/moving discriminating unit 122-4 checks whether or not 
the pixels at the same position of the frame #n'2 and the frame #n*l are still. If the 
pixels are decided to be still, the program moves to step SI 29 where the still/moving 
discriminating unit 122-3 decides whether or not the pixels at the same position of the 
frame #n- 1 and the frame #n are moving. 

If, at step S 129, the pixels at the same position of the frame #n- 1 and the pixel 
of the frame #n are decided to be moving, the program moves to step SI 30 where the 
area decision unit 123-3 sets "1" in the covered background area decision flag 
corresponding to the pixel of the area being verified for indicating that the pixel 
belongs to the covered background area. The area decision unit 123-3 sends the 
covered background area decision flag to the decision flag storage memory 124. The 
program then moves to step S131. 

If at step SI 28 the pixels at the same position of the frame #n'2 and the frame 
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#n-2 are decided to be moving or if at step S 129 the pixels at the same position of the 
frame #n- 1 and the frame #n are decided to be still, the pixel of the frame #n does not 
belong to the covered background area, so the processing at stepS 130 is skipped and 
the program moves to step S131. 

At step S131, the still/moving discriminating unit 122-2 checks whether or not 
the pixels at the same position of the frame #n and the frame #n+l are still. If the 
pixels are decided to be moving, the program moves to step SI 32 where the 
still/moving discriminating unit 122-1 decides whether or not the pixels at the same 
position of the frame #n+l and the frame #n+2 are moving. 

If, at step S132, the pixels at the same position of the frame #n+l and the pixel 
of the frame #n+2 are decided to be still, the prograin moves to step S133 where the 
area decision unit 123-1 sets "1" in the uncovered background area decision flag 
corresponding to the pixel of the area being verified for indicating that the pixel 
belongs to the uncovered background area. The area decision unit 123-1 sends the 
uncovered background area decision flag to the decision flag storage memory 124. 
The program then moves to step SI 34. 

If at step S 131 the pixels at the same position of the frame #n and the frame 
#n+l are decided to be still or if at step SI 32 the pixels at the same position of the 
frame #n+l and the frame #n+2 are decided to be moving, the pixel of the frame #n 
does not belong to the uncovered background area, so the processing at stepS133 is 
skipped and the program moves to step SI 34. 
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At step SI 34, the area specifying unit 103 checks whether or not the area has 
been specified for the totahty of the pixels of the frame #n. If it is decided that the 
area has not been specified for the totahty of the pixels of the frame #n, the program 
reverts to step S 122 to repeat the area specifying processing for the remaining pixels. 

If it is decided at step SI 34 that the area has been specified for the totality of 
the pixels of the frame #n, the program moves to step SI 35 where the synthesis unit 
125 generates the area infomiation indicating the mixed area based on the uncovered 
background area decision flag and the covered background area decision flag, stored 
in the decision flag storage memory 124, while also generating the area infomiation 
indicating to which of the uncovered background area, still area, moving area and the 
uncovered background area belongs each pixel. The synthesis unit 125 sets the 
generated area information in the decision flag storage frame memory 126 to finish the 
processing. 

In this manner, the area specifying unit 103 is able to generate the area 
infonnation, for each of pixels comprehended in a frame, indicating that the pixel in 
question belongs to the movement area, still area, covered background area or to the 
uncovered background area. 

It is also possible for the area specifying unit 103 to apply logical smn to area 
information corresponding to the uncovered background area and the covered 
background area to generate the area infoniiation comprising a flag indicating that a 
given pixel contained in the frame belongs to the movement area, still area or to the 
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mixed area, for each pixel contained in the frame. 

If the object associated with the foreground includes a texture, the area 
specifying unit 103 is able to specify the movement area more accurately. 

The area specifying unit 103 is able to output the area infomiation indicating 
the movement area as the area infonnation indicating the foreground area, or the area 
infomiation indicating the still area as the area infonnation indicating the background 
area. 

In the foregoing, it is assumed that the object corresponding to the background 
is still. However, the above-described area specifying processing can be applied even 
if the picture associated with the background area contains motion. For example, if 
the picture corresponding to the background area is moving unifonnly, the area 
specifying unit 103 shifts the entire picture in association with the movement to 
perfonn the processing in the same way as when the object corresponding to the 
background is still. If the picture associated with the background area contains 
different movements from one location to another, the area specifying unit 103 selects 
the pixels corresponding to the movement to perfonn the above processing. 

Fig.37 shows a block diagram showing an illustrative structure of the mixing 
ratio calculating unit 104. The estimated mixing ratio processor 201 calculates the 
estimated mixing ratio, from one pixel to another, by calculations corresponding to the 
model of the covered background area, based on the input picture, to route the 
calculated estimated mixing ratio to a mixing ratio decision unit 203. 
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An estimated mixing ratio processing unit 202 calculates the estimated mixing 
ratio, from pixel to pixel, by calculations corresponding to the model of the uncovered 
background area, based on the input picture, to route the calculated mixing ratio to the 
mixing ratio decision unit 203. 

Since the object corresponding to the foreground may be assumed to be moving 
at an equal speed within the shutter time, the mixing ratio Oi of a pixel belonging to the 
mixed area has the following properties: That is, the mixing ratio a is changed linearly 
relative to changes in the pixel positions. If the changes of the pixel positions are one- 
dimensional, the changes in the mixing ratio a can be represented as a plane. 

Since the one-frame period is short, it may be assumed that the object 
corresponding to the foreground is a rigid member and is moving at an equal speed. 

Meanwhile, the tilt of the mixing ratio a is inversely proportionate to the 
movement quantity v of the foreground within the shutter time. 

Fig.38 shows an example of an ideal mixing ratio a. The tilt 1 in the mixing 
area with an ideal mixing ratio a can be represented as a reciprocal of the movement 
quantity v. 

In the embodiment of Fig. 39, the pixel value C06 of the seventh pixel from left 
of the frame #n can be represented, using the pixel value P06 of the seventh pixel from 
left of the frame #n-l, by the equation (14): 
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C06= B06/V+B06/V+ F01/v+ F02/v 
= P06/V+ P06/V+ F01/V+ F02/V 

2 

= 2/v-P06+^ i^/v 

•••(14) 

In the equation (14), the pixel value C06 is expressed as a pixel value M of the 
pixel of the mixed area, whilst the pixel value P06 is expressed as a pixel value B of 
the pixel of the background area. That is, the pixel value M of the mixed area and the 
pixel value B of the backgroimd may be represented by the equations (15) and (16), 
respectively: 

M = C06 

•••(15) 

C = a-P+f 

- (16). 

In the equation (14), 2/v corresponds to the mixing ratio a. Since the 
movement quantity v is 4, the mixing ratio a of the seventh pixel from left of the 
frame #n is 0.5. 

By assuming that the pixel value C of the frame #n under consideration and the 
pixel value P of the frame #n'l directly previous to the frame #n as being the pixel 
value of the mixed area and the pixel value of the background, respectively, the 
equation (13) indicating the mixing ratio a can be rewritten to the following equation 
(17): 



C = a-P + f 

•••(17) 

where f denotes the sum SiFi/v of the foreground components contained in the 
considered pixel. There are two variables in the equation ( 1 7), namely the mixing ratio 
a and the sum f of the foreground components. 

Fig.40 shows a model in which the movement quantity v in the uncovered 
background area is 4 and the number of times of the virtual splitting along the time 
axis is 4, with the pixels being shown developed along the time axis direction. 

By assuming, in the uncovered background area, that the pixel value C of the 
frame #n under consideration and the pixel value P of the frame #n+l next to the 
frame #n as being the pixel value of the mixed area and the pixel value of the 
background, respectively, as in the covered background area, discussed above, the 
equation (13) indicating the mixing ratio a can be represented as in the following 
equation (18): 

C = (X-N + f 

•••(18). 

Although the background object is assumed to be still in the foregoing 
description, the equations (14) to (18) may be apphed by exploiting the pixel values 
of the pixels associated with the background movement quantity v even if the 
background object is moving. For example, if, when the movement quantity v of the 
object corresponding to the background is 2 and the number of times of the virtual 
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splitting is 2, the object corresponding to the background is moving towards right in 
the drawing, the pixel value B of the pixel of the background area in the equation (16) 
is the pixel value P04. 

Since the equation ( 1 7) and ( 1 8) each contain two variables, the mixing area a ' 
cannot be found directly. It should be noted that, since the picture in general exhibits 
strong spatial correlation, the pixels proximate to each other are of approximately the 
same pixel values. 

Since the foreground components exhibits strong spatial correlation, the 
equation is modified so that the mixing area a by the sum of the foreground 
components will be derived from the previous frame or the subsequent frames. 

The pixel value Mc of the seventh pixel from left of the frame #n of Fig. 4 1 can 
be represented by the following equation (19): 

M = -• 506+ y F/7v 

where 2/v of the first tenii of the right side corresponds to the mixing ratio Ot. By 
exploiting the pixel value of the subsequent frame #n+l, the second tenn of the right 
side of the equation (19) may be represented by the equation (20): 

12 110 

It is here assumed, by exploiting the spatial correlation of the foreground 
components, the following equation (21) holds: 
F = F05 = F06 = F07 = F08 = F09 = FIO = Fl 1 - F12 



•••(21) 

which may be used to rewrite the equation (20) to 

- (22). 

As a result, P can be represented by the following equation (23) 
P = 22/4 

- (23). 

In general, if it is assumed that the foreground components relevant to the 
mixed area are equal, as shown by the equation (21), the following equation (24): 
P-l-a 

-(24) 

holds, by the ratio of the internal division, for the totality of pixels of the mixed area. 
If the equation (24) holds, the equation ( 1 7) can be expanded as in the equation 

(25): 

C=a-P+ f 

r+v-i 

Suni equation (24) holds, the equation (18) can be expanded as in 

the equation (26): 
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C= a-N+ f 



- (26). 



In the equations (25) and (26), since C, N and P are known pixel values, the 
there is only one variable contained in the equations (25) and (26), that is the mixing 
ratio a. The relation among C, N and P in the equations (25) and (26) is shown in 
Fig.42. It is noted that C, N and P are a pixel value of a pixel of the frame #n under 
consideration, a pixel value of a pixel of the frame #n+l, the position of which in the 
spatial direction is in register with that of the considered pixel, and a pixel value of the 
pixel of the frame #n+l , the position of which in the spatial direction is in register with 
that of the considered pixel, respectively. 

Thus, each one variable is contained in each of the equations (25) and (26), so 
the mixing ratio Ct can be calculated by exploiting the pixel values of the pixels of the 
three frames. The condition for the correct mixing ratio a to be calculated by solving 
the equations (25) and (26) is that the foreground components relevant to the mixed 
area are equal, that is that the pixel values of a number of the consecutive pixels twice 
the movement quantity x, which pixels are in the picture object of the foreground 
imaged in a standstill state, and which are positioned at a boundary of the picture 
object in association with the moving direction of the foreground are constant. 

The mixing ratio a of the pixels belonging to the covered background area is 
calculated by the equation (27), whilst the mixing ratio OC of the pixel belonging to the 
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uncovered background area is calculated by the following equations (27) and (28): 
a = (C-N)/(P-N) 

- (27) 

a = (C-P)/(N-P) 

- (28). 

In Fig.43, which is a block diagram showing the structure of the estimated 
mixing ratio processor 20 1 , a frame memory 22 1 stores the input pictures on the frame 
basis, and feeds a frame, next to the frame being input as an input picture, to a frame 
memory 222 and to a mixing ratio calculating unit 223. 

The frame memory 222 stores the input pictures on the frame basis and routes 
a frame next following the frame being supplied from the frame memory 221 to the 
mixing ratio calculating unit 223. 

So, if the frame #n+l is being input as an input picture to the mixing ratio 
calculating unit 223, the frame memory 221 routes the frame #n to the mixing ratio 
calculating unit 223, whilst the frame memory 222 routes the frame #n' 1 to the mixing 
ratio calculating unit 223. 

The mixing ratio calculating unit 223 calculates an estimated mixing ratio of the 
considered pixel, by calculations of the equation (27) on the pixel value C of the pixel 
of the frame #n under consideration, the pixel value of the pixel of the frame #n+ 1 , the 
spatial position of which is in registration with that of the considered pixel, and the 
pixel value of the pixel of the frame #n'l, the spatial position of which is in 
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registration with that of the considered pixel, and outputs the so-calculated estimated 
mixing ratio. For example, if the background is at a standstill, the mixing ratio 
calculating unit 223 calculates the estimated mixing ratio of the considered pixel, from 
the pixel value C of the pixel of the frame #n under consideration, the pixel value N 
of the pixel of the frame #n+l, the position of which in the frame is the same as that 
of the considered pixel, and the pixel value P of the pixel of the frame #n-l, the 
position of which in the frame is the same as that of the considered pixel, and outputs 
the so-calculated estimated mixing ratio. 

In this manner, the estimated mixing ratio processor 201 calculates the 
estimated mixing ratio, based on the input picture, to route the so-calculated estimated 
mixing ratio to the inixing ratio decision unit 203. 

The estimated mixing ratio processor 202 is similar to the estimated mixing 
ratio processor 20 1 except that the estimated mixing ratio processor 20 1 calculates the 
estimated mixing ratio of the considered pixel in accordance with the equation (27), 
whilst the estimated mixing ratio processor 202 calculates the estimated mixing ratio 
of the considered pixel in accordance with the equation (28), and hence the 
corresponding description is omitted for clarity. 

Fig. 44 shows an example of the estimated mixing ratio calculated by the 
estimated mixing ratio processor 201. Fig.44 shows the estimated mixing ratio for the 
movement quantity v of the foreground corresponding to an object moving at a 
constant speed equal to 1 1 for one line. 
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It is seen that the estimated mixing ratio is changing in the mixed area 
substantially linearly as shown in Fig.38. 

Reverting to Fig.37, the mixing ratio decision unit 203 sets the mixing ratio a 
based on the area information from the area specifying unit 103 indicating to which 
of the foreground area, background area, covered background area and the uncovered 
background area belongs the pixel supphed from the area specifying unit 103 as basis 
for calculation of the mixing ratio a. The mixing ratio decision unit 203 sets 0 or 1 as 
the mixing ratio if the pixel as a basis for calculation belongs to the foreground area 
or to the background area, respectively. On the other hand, the mixing ratio decision 
unit 203 sets the estimated mixing ratio supplied from the estimated mixing ratio 
processor 20 1 as the mixing ratio a if the pixel as a basis for calculation belongs to the 
covered background area, while setting the estimated mixing ratio supplied from the 
estimated mixing ratio processor 202 as the mixing ratio a if the pixel as a basis for 
calculation belongs to the uncovered background area. The e203 outputs the mixing 
ratio a as set based on the area infomiation. 

In Fig.45, which is a block diagram showing an alternative structure of the 
mixing ratio calculating unit 104, a selection unit 23 1 routes the pixel belonging to the 
covered background area and pixels of the associated previous and subsequent frames 
to an estimated mixing ratio processor 232, based on the area information supplied 
from the area specifying unit 103. The selection unit 23 1 routes the pixels belonging 
to the uncovered background area and pixels of the associated previous and 
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subsequent frames to an estimated mixing ratio processor 233, based on the area 
infonnation supplied from the area specifying unit 103. 

The estimated mixing ratio processor 232 calculates the estimated mixing ratio 
of the considered pixel belonging to the covered background area, by calculations in 
accordance with the equation (27), based on the pixel values input from the selection 
unit 231, to route the so-calculated estimated mixing ratio to a selection unit 234. 

The estimated mixing ratio processor 233 calculates the estimated mixing ratio 
of the considered pixel belonging to the imcovered background area, by calculations 
in accordance with the equation (28), based on the pixel values input from the 
selection unit 23 1, to route the so-calculated estimated mixing ratio to a selection unit 
234. 

The selection unit 234 sets the mixing ratio OC based on the area information 
from the area specifying unit 103 indicating to which of the foreground area, 
background area, covered background area and the uncovered background area 
belongs the pixel supplied from the area specifying unit 103 as basis for calculation 
of the mixing ratio a. The mixing ratio decision unit 203 sets 0 or 1 as the mixing ratio 
if the pixel as a basis for calculation belongs to the foreground area or to the 
background area, respectively. On the other hand, the selection unit 234 sets the 
estimated mixing ratio supphed from the estimated mixing ratio processor 232 as the 
mixing ratio a if the pixel as a basis for calculation belongs to the covered background 
area, while setting the estimated mixing ratio supplied from the estimated mixing ratio 
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processor 233 as the mixing ratio a if the pixel as a basis for calculation belongs to the 
uncovered background area. The selection unit 234 outputs the mixing ratio a 
selected and set based on the area infonnation. 

The mixing ratio calculating unit 104, having a modified structure shown in 
Fig.45, calculates the mixing ratio OC, from one pixel of the picture to another, to 
output the calculated mixing ratio OL. 

Referring to the flowchart of Fig.46, the processing for calculating the mixing 
ratio a of the mixing ratio calculating unit 104, the configuration of which is shown 
in Fig. 3 7, is explained. At step S151, the mixing ratio calculating unit 104 acquires 
the area information supplied from the area specifying unit 103. At step S151, the 
mixing ratio calculating unit 104 acquires the area infonnation supplied from the area 
specifying unit 1 03 . At step S 1 52, the estimated mixing ratio processor 201 calculates 
the estimated mixing ratio by a model corresponding to the covered background area 
to route the so-calculated estimated mixing ratio to the mixing ratio decision unit 203 . 
The processing for calculating the estimated mixing ratio will be explained 
subsequently in detail by referring to flowchart of Fig.47. 

At step S 1 53, the estimated mixing ratio processor 202 calculates the estimated 
mixing ratio by a model corresponding to the covered background area to route the 
so-calculated estimated mixing ratio to the mixing ratio decision unit 203. 

At step SI 54, the mixing ratio calculating unit 104 checks whether or not the 
mixing ratio CL has been estimated for the entire frame. If it is found that the mixing 
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ratio a has not been estimated for the entire frame, the program reverts to step SI 52 
to execute the processing of estimating the mixing ratio a for the next pixel. 

If it is decided at step S154 that the mixing ratio CL has been estimated for the 
entire frame, the program reverts to step S 1 5 5 where the mixing ratio decision unit 203 
sets the mixing ratio OL based on the area information supphed from the area specifying 
unit 103 and which indicates to which of the foreground area, background area, 
covered background area or the uncovered background area belongs the pixel. The 
mixing ratio decision unit 203 sets 0 or 1 as the mixing ratio if the pixel as a basis for 
calculation belongs to the foreground area or to the background area, respectively. On 
the other hand, the mixing ratio decision unit 203 sets the estimated mixing ratio 
supplied from the estimated mixing ratio processor 201 as the mixing ratio a if the 
pixel as a basis for calculation belongs to the covered background area, while setting 
the estimated mixing ratio supplied from the estimated mixing ratio processor 202 as 
the mixing ratio CL if the pixel as a basis for calculation belongs to the uncovered 
background area. The processing then is finished. 

In this manner, the mixing ratio calculating unit 104 is able to calculate the 
mixing ratio CL, as a characteristic value for each pixel, based on the area information 
supplied from the area specifying unit 103 and on the input picture. 

The processing for calculating the mixing ratio CL by the mixing ratio calculating 
unit 104 shown in Fig.45 is similar to that explained by referring to the flowchart of 
Fig.46 and hence is not explained specifically. 
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Referring to the flowchart of Fig.47, the processing similar to step SI 52 of 
Fig.46 for estimating the mixing ratio by a model corresponding to the covered 
background area is explained. 

At step SI 71, the mixing ratio calculating unit 223 acquires the pixel value C 
of the considered pixel of the frame #n from the frame memory 221. 

At step S172, the mixing ratio calculating unit 223 acquires the pixel value C 
of the considered pixel of the frame #n-l from the frame memory 222. 

At step 8173, the mixing ratio calculating unit 223 acquires the pixel value N 
of the frame #n+l, corresponding to the considered pixel contained in the input 
picture. 

At step SI 74, the mixing ratio calculating unit 223 calculates the estimated 
mixing ratio based on the pixel value C of the considered pixel of the frame #n, pixel 
value P of the pixel of the frame #n-l and on the pixel value N of the pixel of the 
frame #n+l. 

At step SI 75, the mixing ratio calculating unit 223 checks whether or not the 
processing for calculating the estimated mixing ratio has been finished for the entire 
frame. If it is decided that the processing for calculating the estimated mixing ratio 
has not been finished for the entire frame, the program reverts to step S171 to repeat 
the processing of calculating the estimated mixing ratio for the next pixel. 

If it is verified at step SI 75 that the processing for calculating the estimated 
mixing ratio has been finished for the entire frame, the processing is fmished. 
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In this manner, the estimated mixing ratio processor 20 1 is able to calculate the 
estimated mixing ratio based on the input picture. 

The processing for estimating the mixing ratio by the model corresponding to 
the uncovered background area at step S153 of Fig.46 is similar to the processing 
exploiting the equation corresponding to the model of the uncovered background area, 
as shown in the flowchart of Fig.47, and hence is not explained specifically. 

Meanwhile, since the estimated mixing ratio processor 232 and the estimated 
mixing ratio processor 233, shown in Fig.45, calculates the estimated mixing ratio by 
executing the processing similar to the processing of the flowchart of Fig.47, the 
corresponding operation is omitted for simplicity. 

In the foregoing explanation, it is assumed that the object corresponding to the 
background is at a standstill. However, the processing for finding the mixing ratio a 
can also be applied to a case in which a picture corresponding to the background 
contains the movement. For example, if a picture corresponding to the background 
area is moving uniformly, the estimated mixing ratio processor 201 shifts the entire 
picture in keeping with the movement of the background to perform the processing as 
if the object corresponding to the background is at a standstill. On the other hand, if 
the picture corresponding to the background area contains movements of the 
background which differ from one location to another, the estimated mixing ratio 
processor 201 selects the pixel associated with the background movement, as the pixel 
corresponding to the pixel belonging to the mixed area, to execute the above-described 
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processing. 

The structure of the mixing ratio calculating unit 104 shown in Figs.37 or 45 is 
merely illustrative. 

It is also possible for the mixing ratio calculating unit 104 to execute only the 
processing for estimating the mixing ratio by the model corresponding to the covered 
background area to output the so-calculated estimated mixing ratio as the mixing ratio 
a. In this case, the mixing ratio a indicates the proportion of the foreground and the 
background for a pixel belonging to the covered background area and for a pixel 
belonging to the uncovered background area, respectively. If the absolute value of the 
difference between the so-calculated mixing ratio a and 1 is calculated to set the so- 
calculated absolute value as the mixing ratio a, the signal processor 12 is able to find 
the mixing ratio a indicating the proportion of the background component for the pixel 
belonging to the uncovered background area. 

It is also possible to execute only the processing for mixing ratio estimation by 
a model corresponding to the uncovered background area for the totality of the pixels 
to output the so-calculated estimated mixing ratio as the mixing ratio a. 

The foreground/background separating unit 105 is now explained. In Fig. 4 8 
which is a block diagram showing the illustrative structure of the 
foreground/background separating unit 105, an input picture, fed to the 
foreground/background separating unit 105, is suppHed to a separating unit 251, a 
switch 252 and to a switch 254. The area information from the area specifying unit 
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103, specifying the inforaiation indicating the covered background area information 
and the infonnation indicating the uncovered background area infonnation, is fed to 
the separating unit 251, whilst the area infonnation indicating the foreground 
infonnation and the area infonnation indicating the background area are routed to the 
switches 252, 254, respectively. 

The mixing ratio a supplied from the mixing ratio calculating unit 104 is sent 
to the separating unit 251. 

The separating unit 251 separates the foreground component from the input 
picture, based on the area infonnation indicating the covered background area 
infonnation, the area infonnation indicating the uncovered background area 
infonnation and the mixing ratio Ot, and supplies the so-separated foreground 
component to a synthesis imit 253, while separating the background component from 
the input picture to route the so-separated background component to the synthesis unit 
255. 

When a pixel corresponding to the foreground is input, the switch 252 is closed, 
based on the area infonnation indicating the foreground area, to route only the pixels 
corresponding to the foreground contained in the input picture to the synthesis unit 
253. 

When a pixel corresponding to the background is input, the switch 254 is 
closed, based on the area infonnation indicating the background area, to route only the 
pixel corresponding to the background contained in the input picture to the synthesis 
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unit 255. 

The synthesis unit 253 synthesizes a foreground component picture, based on 
the component from the separating unit 251, corresponding to the foreground, and on 
the pixel from the switch 252, corresponding to the foreground, to output the 
synthesized foreground component picture. Since the foreground area and the mixed 
area are not overlapped, the synthesis unit 253 apphes the processing of logical sum 
to the component corresponding to the foreground and to the pixel corresponding to 
the foreground to synthesize the foreground component picture. 

In the initializing processing, executed first in the processing for synthesizing 
the foreground component picture, the synthesis unit 253 stores a picture with all zero 
pixel values in an internal frame memoiy to store (overwrite) the foreground 
component picture in the processing for synthesizing the foreground component 
picture. Thus, in the pixels corresponding to the background area, in the foreground 
component picture output by the synthesis unit 253, Os are stored as pixel values. 

The synthesis unit 255 synthesizes the background component picture, based 
on the components from the separating unit 251 and on the pixels from the switch 254 
corresponding to the background, to output the synthesized background component 
picture. Since the background area and the mixed area are not overlapped, the 
synthesis unit 255 applies the processing of logical smn to the component 
corresponding to the background and to the pixel corresponding to the background to 
synthesize the background component picture. 
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In the initializing processing, executed first in the processing for synthesizing 
the background component picture, the synthesis unit 255 stores a picture with all zero 
pixel values in an internal fi-ame memory to store (overwrite) the background 
component picture in the processing for synthesizing the background component 
picture. Thus, in the pixels corresponding to the foreground area, in the background 
component picture output by the synthesis unit 255, Os are stored as pixel values. 

Fig.49 shows an input picture, fed to the foreground/background separating unit 
105, and the foreground component picture and the background component picture, 
output firom the foreground/background separating unit 105. 

Fig.49A schematically shows the displayed picture, while Fig.49B 
diagrammatically shows a model obtained on developing one-line pixels comprised of 
pixels belonging to the foreground area, pixels belonging to the background area and 
the pixels in the mixed area, along the tiine axis. 

Referring to Figs.49 A and 49B , the background component picture, output from 
the foreground/background separating unit 105, is comprised of a background 
component contained in the pixels belonging to the background area and pixels 
belonging to the mixed area. 

Referring to Figs.49A and 49B, the foreground component picture, output from 
the foreground/background separating unit 105, is comprised of a foreground 
component contained in the pixels belonging to the foreground area and pixels 
belonging to the mixed area. 
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The pixel values of the pixels of the mixed area are separated by the 
foreground/background separating unit 105 into the background component and the 
foreground' component. The background component, thus separated, makes up the 
background component picture along with the pixels belonging to the background area. 
The foreground component separated makes up the foreground component picture 
along with the pixels belonging to the foreground area. 

In this manner, the pixel values of the pixels of the foreground component 
picture, associated with the background area, are set to 0, while meaningful pixel 
values are set in the pixels corresponding to the foreground area and to the pixels 
corresponding to the mixed area. Similarly, the pixel values of the pixels of the 
background component picture, associated with the foreground area, are set to 0, while 
meaningful pixel values are set in the pixels corresponding to the background area and 
to the pixels corresponding to the mixed area. 

The processing executed by the separating unit 25 1 in separating the foreground 
and background components from the pixels belonging to the mixed area is 
explained. 

Fig. 50 diagrammatically shows a model of a picture showing foreground and 
background components of two frames including the foreground corresponding to an 
object moving from left to right in the drawing. In the picture model of Fig. 50, the 
movement quantity v of the foreground is 4, with the number of times of the virtual 
splitting being 4. 
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In the frame #n, the leftmost pixel and fourteenth to eighteenth pixels from left 
are composed only of the background components and belong to the background area. 
In the frame #n, the second to fourth pixels from left include background and 
foreground components and belong to the uncovered background area. In the frame 
#n, the eleventh to thirteenth pixels from left include background and foreground 
components and belong to the covered background area. In the frame #n, the fifth to 
tenth pixels from left include only foreground components and belong to the covered 
foreground area. 

In the frame #n+l, the first to fifth pixels and the eighteenth pixel from left 
are composed only of the background component and belong to the background area. 
In the frame #n+l, the sixth to eighth pixels from left include background and 
foreground components and belong to the uncovered background area. In the frame 
#n+l , the fifteenth to seventeenth pixels from left include background and foreground 
components and belong to the covered background area. In the frame #n+ 1 , the ninth 
to fourteenth pixels from left include only foreground components and belong to the 
covered foreground area. 

Fig. 5 1 illusfrates the processing for separating the foreground component from 
the pixels belonging to the covered background area. In Fig.51, al to al8 represent 
the mixing ratio values associated with respective pixels of the frame #n. In Fig.51, 
the fifteenth to seventeenth pixels from left belong to the covered background area. 

The pixel value C15 which is the fifteenth pixel from left of the frame #n is 
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represented by the following equation (29): 
CI 5 = B 1 5/v + F09/V + F08/v + F07/v 

= al5-B15 + F09/V + F08/V + F07/v 
= al5-P15 + F09/V + F08/v + F07/v 

- (29) 

where al5 is the mixing ratio of the fifteenth pixel from left of the frame #n and P15 
is the pixel value of the fifteenth pixel from left of the frame #n-l. 

Based on the equation (29), the sum fl5 of the foreground components of the 
fifteenth pixel of the frame #n is represented by the equation (30): 

f 1 5 = F09/V + FOS/v + F07/V 
= C15"al5-P15 



Similarly, the sum fl6 of the foreground components of the sixteenth pixel of 
the frame #n and the sum first wiring pattern 17 of the foreground components of the 
seventeenth pixel of the frame #n are represented by the equations (31) and (32): 

fl6 = C16-al6-P16 



- (30). 



•••(31) 



and 



fl7 = C17^al7-P17 



- (32) 



respectively. 
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In this manner, the foreground components fc, contained in the pixel value C 
of the pixel belonging to the covered background area, may be calculated by the 
equation (33): 

fc-C-Ci-P 

- (33). 

Fig. 52 illustrates the processing for separating the foreground components from 
the pixels belonging to the uncovered background area. In Fig.52, (X 1 to oc 1 8 represent 
the values of the mixing ratio for respective pixels of the frame #n. In Fig.52, the 
second to fourth pixels from left belong to the uncovered background area. 

The pixel values C02 of the second pixel from left of the frame #n are 
represented by the equation (34): 

C02 = B02/V + B02/V + B02/v + FOl/v 
= a2-B02 + F01/v 
= a2-N02 + F01/v 

- (34) 

where dl is a mixing ratio of the second pixel from left of the frame #n and N02 is a 
pixel value of the second pixel from left of the frame #n+l. 

Based on the equation (34), the sum fD2 of the foreground component of the 
second pixel from left of the frame #n is represented by the equation (35): 

f02 = F01/v 

= C02-a2-NO2 
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- (35) 

Similarly, the sum fD3 of the foreground components of the third pixel from left 
of the frame #n and the sum fD4 of the foreground components of the fourth pixel 
from left of the frame #n are represented by the following equations (36) and (37): 

fD3 = C03-Ci3'NO3 

-(36) 

and 

fD4 = C04-0C4-NO4 

-(37) 

respectively. 

The foreground component fu included in the pixel value C of the pixel 
belonging to the uncovered background area is calculated by the following equation 
(38): 

fri = C-a-N 

-(38) 

where N is a pixel value of a corresponding pixel of the next frame. 

In this manner, the separating unit 251 is able to separate the foreground 
components and the background components from the pixels belonging to the mixed 
area, based on the information indicating the covered background area and on the 
infonuation indicating the covered background area, contained in the area infomiation, 
and on the pixel-based mixing ratio CL. 
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In Fig.53, which is a block diagram showing an illustrative structure of the 
separating unit 25 1 adapted for executing the above-described processing, a picture 
input to the separating unit 251 is input to the frame memory 301, whilst the area 
infoniiation indicating the covered background area and the uncovered background 
area, and the mixing ratio a, are input to a separating processing block 302. 

The fraine memory 301 stores the input pictures on the frame basis. If the 
object of processing is the frame #n, the frame memory 301 stores the frame #n'l, 
directly previous to the frame #n, frame #n and the frame #n+l next to the frame #n. 

The frame memory 301 routes the corresponding pixels of the frame #n-l, 
frame #n and the frame #n+l, to a separating processing block 302. 

Based on the area information indicating the covered background area 
infonnation and the uncovered background area infonnation, and on the mixing ratio 
a, the separating processing block 302 perfonns calculations, explained with reference 
to Figs. 51 and 52, on the pixel values of the corresponding pixels of the frame #n-l, 
frame #n and the frame #n+l, supphed from the frame memory 301, to separate the 
foreground and background components from the pixels belonging to the mixed area 
of the frame #n to route the separated components to a frame memory 303. 

The separating processing block 302 is made up of an uncovered area processor 
3 11, a covered area processor 312, a synthesis unit 313 and a synthesis unit 3 14. 

The uncovered area processor 3 1 1 includes a multiplier 32 1 which multiplies 
the pixel value of the pixel of the frame #n+l supplied from the frame memory 301 
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with the mixing ratio a to route the resulting product to the switch 322, which is 
closed when the pixel of the fraine #n (corresponding to the pixel of the frame #n+l) 
supplied from the frame memory 30 1 is in the uncovered background area, to route the 
pixel value multiplied with the mixing ratio 0£ sent from the multiplier 321 to an 
operating unit 322 and to the synthesis unit 314. The pixel value of the pixel of the 
frame #n+l from the switch 322, multiplied by the mixing ratio a, is equal to the 
background component of the pixel value of the corresponding pixel of the frame #n. 

An operating unit 323 subtracts the background component supplied from the 
switch 322 from the pixel value of the pixel of the frame #n supplied from the frame 
memory 301 to find the foreground component. The operating unit 323 routes the 
foreground component of the pixel of the frame #n, belonging to the uncovered 
background area, to the synthesis unit 313. 

The covered area processor 3 1 2 includes a multiplier 33 1 which multiplies the 
pixel value of the pixel of the frame #n* 1 supplied from the frame memory 301 with 
the mixing ratio a to route the resulting product to the switch 322, which is closed 
when the pixel of the frame #n (corresponding to the pixel of the frame #n- 1 ) supplied 
from the frame memory 30 1 is in the covered background area, to route the pixel value 
multiphed with the mixing ratio a sent from the multiplier 33 1 to an operating unit 333 
and to the synthesis unit 314. The pixel value of the pixel of the frame #n' 1 from the 
switch 332, multiplied by the mixing ratio a, is equal to the background component 
of the pixel value of the corresponding pixel of the frame #n. 
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An operating unit 333 subtracts the background component supplied from the 
switch 332 from the pixel value of the pixel of the frame #n supplied from the frame 
memory 301 to find the foreground component. The operating unit 333 routes the 
foreground component of the pixel of the frame #n, belonging to the covered 
background area, to the synthesis unit 313. 

The synthesis unit 313 synthesizes the foreground component of the pixel from 
the operating unit 323, belonging to the uncovered background area, to the foreground 
component of the pixel belonging to the covered background area, to route the 
resulting sum to the frame memory 303. 

The synthesis unit 314 synthesizes the background component of the pixel 
from the operating unit 323, belonging to the uncovered background area, to the 
background component of the pixel from the switch 332 belonging to the covered 
background area, to route the resulting sum to the frame memory 303. 

The frame memory 303 stores the foreground and background components of 
the pixels of the mixed area of the frame #n, supplied from the separating processing 
block 302. 

The frame memory 303 outputs the stored foreground component of the pixels 
of the mixed area of the frame #n and the stored background component of the pixels 
of the mixed area of the frame #n. 

By exploiting the mixing ratio Cd as a characteristic value, the foreground and 
background components contained in the pixel value can be separated completely from 
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each other. 

The synthesis unit 253 synthesizes the foreground component of the pixels of 
the mixed area of the frame #n , output by the separating unit 251, and the pixels 
belonging to the foreground area, to each other, to generate a foreground component 
picture. The synthesis unit 255 synthesizes the background component of the pixels 
of the mixed area of the frame #n , output by the separating unit 251, and the pixels 
belonging to the background area, to each other, to generate a background component 
picture. 

Fig. 54 shows an example of a foreground component picture and an example 
of the background component picture corresponding to the frame #n of Fig.50. 

Fig. 54 A shows an example of the foreground component picture corresponding 
to the frame #n of Fig.50. The pixel values of the leftmost pixel and the fourteenth 
pixel from left were composed only of the background component before foreground/ 
background separation, and hence are equal to 0. 

The second to fourth pixels from left belonged to the uncovered background 
area before foreground/ background separation, with the background component being 
0 and with the foreground component being left intact. The eleventh to thirteenth 
pixels from left belonged to the covered background area before foreground/ 
background separation, with the background component being 0 and with the 
foreground component being left intact. The fifth to tenth pixels from left are left 
intact because these pixels are composed only of the background components. 
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Fig.54B shows an example of a background component picture corresponding 
to the frame #n of Fig. 50. The leftmost pixel and the fourteenth pixel from left are 
composed only of the background component before foreground/ background 
separation, and hence are left intact. 

The second to fourth pixels from left belonged to the uncovered background 
area before foreground/ background separation, with the foreground component being 
0 and with the background component being left intact. The eleventh to thirteenth 
pixels from left belonged to the covered background area before 
foreground/background separation, with the background component being 0 and with 
the foreground component being left intact. The pixel values of the fifth to tenth 
pixels from left are set to zero because these pixels are composed only of the 
foreground components. 

Referring to the flowchart of Fig. 55, the processing for foreground/background 
separation by the foreground/background separating unit 105 is explained. At step 
S20 1 , the frame memory 30 1 of the separating unit 25 1 acquires an input picture and 
stores the frame #n, to be processed for foreground/background separation, along with 
the previous frame #n'l and the subsequent frame #n+l. 

At step S202, the separating processing block 302 of the separating unit 251 
acquires the area inforaiation supplied from the mixing ratio calculating unit 104. At 
step S203, the separating processing block 302 of the separating unit 25 1 acquires the 
mixing ratio « routed from the mixing ratio calculating unit 1 04. 
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At step S204, the uncovered area processor 311 extracts the background 
component, based on the area infonnation and the mixing ratio Cd, the pixel values of 
pixels belonging to the uncovered background area, supplied from the frame memory 
301. 

At step S205, the uncovered area processor 311 extracts the foreground 
component, based on the area infonnation and the mixing ratio a, the pixel values of 
pixels belonging to the uncovered background area, supplied from the frame memory 
301. 

At step S206, the covered area processor 312 extracts the background 
component, based on the area infonnation and the mixing ratio a, the pixel values of 
pixels belonging to the covered background area, supplied from the frame memoiy 
301. 

At step S207, the covered area processor 312 extracts the foreground 
component, based on the area infonnation and the mixing ratio a, the pixel values of 
pixels belonging to the covered background area, supplied from the frame memory 
301. 

At step S208, the synthesis unit 313 synthesizes the foreground component, 
belonging to the uncovered background area, extracted by the processing of step S205 , 
and the foreground component, belonging to the covered background area, extracted 
by the processing of step S207. The synthesized foreground component is routed to 
the synthesis unit 253, which then synthesizes the pixels belonging to the foreground 
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area supplied via switch 252 and the foreground component supphed form the 
separating unit 25 1 to generate a foreground component picture. 

At step S209, the synthesis unit 3 14 synthesizes the background component, 
belonging to the uncovered background area, extracted by the processing of step S204, 
and the background component, belonging to the covered background area, extracted 
by the processing of step S206. The synthesized foreground component is routed to 
the synthesis unit 255, which then synthesizes the pixels belonging to the foreground 
area supplied via switch 254 and the background component supplied forai the 
separating unit 25 1 to generate a background component picture. 

At step S2 1 0, the synthesis unit 253 outputs the foreground component picture. 
At step S211, the synthesis unit 255 outputs the background component picture to 
tenninate the processing. 

In this maimer, the foreground/background separating unit 105 is able to 
separate the foreground component and the background component from the input 
picture, based on the area infonnation and the mixing ratio a, to output a foreground 
component picture, made up only of the foreground components, and the background 
component picture, made up only of the background components. 

The adjustment of the quantity of the motion blurring from the foreground 
component picture is explained. 

In Fig.56, which is a block diagram showing an illustrative structure of the 
motion blurring adjustment unit 106, the motion vector supplied from the motion 
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detection unit 102, the corresponding position infonnation and the area information 
suppHed from the area specifying unit 1 03 are routed to a processing unit decision unit 
35 1 and to a modeUingunit 352. The foreground component picture suppHed from the 
foreground/background separating unit 105 is sent to an addition unit 354. 

The processing unit decision unit 35 1 routes the generated processing unit to 
the modelling unit 352, along with the motion vector, based on the motion vector, the 
corresponding position infonnation and the area infonnation. 

The processing unit, generated by the processing unit decision unit 351, 
represents consecutive pixels, beginning from a pixel corresponding to the covered 
background area of the foreground component picture and extending up to a pixel 
corresponding to the uncovered background area, along the movement direction, or 
consecutive pixels, beginning from a pixel corresponding to the uncovered background 
area and extending up to a pixel corresponding to the covered background area, along 
the movement direction, as shown for example in Fig. 57. The processing unit is made 
up e.g., of an upper left point and a lower right point. The upper left point is the 
position of a pixel specified by the processing unit and lying at the leftmost or 
uppermost point on a picture. 

The modelling unit 352 executes the modelling based on the motion vector and 
on the input processing unit. More specifically, the modelling unit 352 may hold at 
the outset plural models corresponding to the number of pixels contained in the 
processing unit, the number of times of the virtual splitting of the pixel values in the 
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time axis direction and the number of the pixel-based foreground components to select 
a model specifying the correspondence between the pixel values and the foreground 
components, based on the processing unit and on the number of times of the virtual 
splitting of pixel values in the time axis direction, as shown in Fig. 5 8. 

For example, with the number of pixels corresponding to a processing unit 
being 1 2 and with the movement quantity v in the shutter time being 5, the modelling 
unit 352 selects a sum total of eight foreground components, with the leftmost pixel 
including one foreground component, the second left pixel including two foreground 
components, the third left pixel including three foreground components, the fourth left 
pixel including four foreground components, the second left pixel including two 
foreground components, the third left pixel including three foreground components, 
the fourth left pixel including four foreground components, the fifth left pixel 
including five foreground components, the sixth left pixel including five foreground 
components, the seventh left pixel including five foreground components, the eighth 
left pixel including five foreground components, the ninth left pixel including four 
foreground components, the tenth left pixel including three foreground components, 
the eleventh left pixel including two foreground components, and the twelfth left pixel 
including one foreground component. 

Instead of selecting from the pre-stored model, the modelling unit 352 may 
generate the model based on the motion vector and on the processing unit, when the 
motion vector and the processing unit are supplied thereto. 
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The modelling unit 352 sends a selected model to an equation generating unit 

353. 

The equation generating unit 353 generates an equation based on the model 
supplied from the modelling unit 352. Referring to the model of the foreground 
component picture, shown in Fig. 5 8, an equation generated by the equation generating 
unit 353 when the number of the foreground components is 8, the number of pixels 
corresponding to the processing unit is 12, the movement quantity v is 5, and the 
number of times of the virtual splitting is 5 is explained. 

When the foreground components corresponding to the shutter time/v contained 
in the foreground component picture are FOl/v to F08/v, the relation between the 
foreground components FO 1/v to F08/v and the pixel values CO 1 to C 12 is represented 
by the equations (39) to (50): 
COl =F01/v 

•••(39) 

C02 = F02/v + F01/v 

•••(40) 

C03 = F03/V + F02/V + FOl/v 

•••(41) 

C04 = F04/V + F03/V + F02/v + FOl/v 

•••(42) 

C05 = F05/V + F04/V + F03/v + F02/v + FOl/v 



C06 = F06/V + F05/V + F04/v + F03/v + F02/v 



C07 = F07/V + F06/V + F05/v + F04/v + F03/v 



COS = F08/V + F07/V + F06/v + F05/v + F04/v 



C09 = F08/v + F07/V + F06/v + F05/v 



CIO = F08/V + F07/V + F06/v 



Cll -F08/V + F07/V 



•••(43) 



•••(44) 



"(45) 



•••(46) 



"(47) 



"•(48) 



"(49) 



C12 = F08/v 

•••(50) 

The equation generating unit 353 modifies the generated equations to generate 
equations. The equations generated by the equation generating unit 353 are indicated 
by the equations (5 1) to (62): 

COl = 1-FOl/v + 0-F02/V + 0-F03/V + 0^F04/v + 0-F05/v + 0-F06/v + 0-F07/v + 
0-F08/V 

•"(51) 
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C02 = 1-FOl/v + 1-F02/V + 0-F03/v + 0-F04/v + 0-F05/v + 0-F06/v + 0-F07/v + 
0-F08/V 

•••(52) 

COS = 1-FOl/v + l^F02/v + l^F03/v + 0-F04/v + 0-F05/v + 0^F06/v + 0-F07/v + ' 
O-FOS/v 

•••(53) 

C04 = l^FOl/v + hF02/v + l-F03/v + l-F04/v + 0^F05/v + 0^F06/v + 0-F07/v + 
0^F08/v 

••■(54) 

C05 = 1-FOl/v + 1-F02/V + l-F03/v + l-F04/v + l-F05/v + 0-F06/v + 0-F07/v + 
O-FOS/v 

•••(55) 

C06 = O-FOl/v + l^F02/v + l-F03/v + l-F04/v + l'F05/v + l-F06/v + 0-F07/v + 
0-F08/V 

•••(56) 

C07 = O^FOl/v + 0^F02/v + l-F03/v + l'F04/v + l^F05/v + l-F06/v + l^F07/v + 
0^F08/v 

•••(57) 

COS = O^FOl/v + 0-F02/V + 0-F03/v + l-F04/v + l^F05/v + 0-F06/v + l-F07/v + 
l^F08/v 



•••(58) 
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C09 = 0-FOl/v + 0-F02/V + 0-F03/v + 0-F04/v + l-F05/v + l-F06/v + l-F07/v + 
1-F08/V 

•••(59) 

CIO = 0-FOl/v + 0-F02/V + 0-F03/v + 0-F04/v + 0-F05/v + l-F06/v + l-F07/v + 
1-F08/V 

•••(60) 

Cll = 0-FOl/v + 0-F02/V + 0-F03/V + 0-F04/v + 0-F05/v + 0-F06/v + 0'F07/v + 
1-F08/V 

•••(61) 

C12 = 0-FOl/v + 0-F02/V + 0-F03/v + l^F04/v + l-F05/v + 0^F06/v + l-F07/v + 
l-FOB/v 

•••(62) 

The equations (51) to (62) may also be represented by the equation (63): 

08 

CJ = Y.a,J-Filv ...^g3j 

where j indicates the pixel position. In this case, j assumes one of values of 1 to 12. 
On the other hand, i denotes the position of the foreground value, and assumes one of 
values of 1 to 8. Aij has values of 0 or 1 in association with the values of i and j. 

If an error is taken into account, the equation (63) may be represented by the 
equation (64): 



- (64) 



- (65). 
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08 

C/= Y, cuij ■ Fi / V + ej 

1=01 

where ej is an error contained in a considered pixel Cj. 

The equation (64) may be rewritten to the equation (65): 

08 

^J= CJ-Y^aij-Fi Iv 

1=01 

In order to apply the least square sum, a square sum of errors E is defined as 
indicated by the equation (66): 

....... 

y=01 (OO)- 

In order to minimize the error, it suffices if the value of the partial 
differentiation by a variable FK with respect to the error square sum E is 0. Fk is found 
to satisfy the equation (67): 

f 

5Fk '^.f^'SFk 

12 08 

= ^-Y.{iCj-Y. aij ■ Fi I v) . (- akj / v) = 0 

y=oi ,=01 ••• (67). 

Since the movement quantity v in the equation (67) is constant, the equation 

(68): 

12 08 

llakj-{Cj-Y^aij-Filv)=Q ^ 

j=oi 1=01 y^'=>) 

can be derived. 



m 
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Developing the equation (68) and shifting the term, we obtain the equation (69): 

12 08 12 

2] (akJ-Y,aiJ- Fi) = V £ akj- Cj 

/=01 ;=01 •'• (69). 

The equation (69) is expanded into eight equations by substituting one of 
integers of 1 to 8. The resulting eight equations can be represented by matrix by a sole 
equation tenned a normal equation. 

An example of the normal equation, generated by the equation generating unit 
353 based on the minimum square method, is the following equation (70): 



"54321000" 


"For 


45432100 


F02 


34543210 


F03 


23454321 


F04 


12345432 


F05 


01234543 


F06 


00123454 


F07 


00012345 


FOB 



05 

- (70). 

If the equation (70) is expressed as A'F = vC, C, A and v are known, while F 
is unknown. On the other hand, A and v are known at the modelling stage, C becomes 
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known on inputting a pixel value in the adding operation. 

By calculating the foreground component by the nonnal equation which is 
based on the least square method, it is possible to effect scattering of errors contained 
in the pixel C. 

The equation generating unit 353 sends the so-generated nonnal equation to an 
addition unit 354. 

Based on the processing unit, supphed from the processing unit decision unit 
351, the addition unit 354 sets the pixel value C, contained in the foreground 
component picture, in the matrix equation supplied from the equation generating unit 
353. The addition unit 354 sends the matrix, having set the pixel value C set therein, 
to an operating unit 355. 

The operating unit 355 calculates the foreground component freed of motion 
blurring Fi/v, by the processing which is based on the solution method, such as 
Gauss -JordAn erasure method, to find Fi corresponding to one of integers 0 to 8 of i 
as pixel values of the foreground freed of the motion blurring to output the foreground 
component picture composed of pixel values freed of motion blurring Fi to a motion 
blurring adding unit 356 and to a selection unit 357. 

Meanwhile, FOl to F08 are set to C03 to CIO, in the foreground component 
picture freed of the motion blurring, shown in Fig. 5 9, in order to produce no changes 
in the position of the foreground component picture relative to the picture screen. An 
arbitrary position can be set. 
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The motion blurring adding unit 356 is able to adjust the quantity of the motion 
blurring by imparting the motion blurring adjustment quantity v' different from the 
movement quantity v, for example, the motion blurring adjustment quantity v' equal 
to one-half the movement quantity v, or the motion blurring adjustment quantity v' 
irrelevant to the movement quantity v, to adjust the value of the motion blurring 
quantity. For example, the motion blurring adding unit 356 divides the pixel value Fi 
of the foreground freed of the motion blurring by the motion blurring adjustment value 
v' to calculate the foreground component Fi/v' and sums the foreground components 
Fi/v' to generate a pixel value adjusted for the motion blurring quantity, as shown in 
Fig. 60. For example, if the motion blurring adjustment quantity v' is 3, the pixel value 
C02 is (FOl/v'), the pixel value C03 is (FOl + F02)/v', the pixel value C04 is (FOl + 
F02 + F03)/v' and the pixel value COS is (F02 + F03 + F04)/v'. 

The motion blurring adding unit 356 sends the foreground component picture, 
adjusted for the motion blurring quantity, to the selection unit 357. 

Based on the selection signal corresponding to the user's selection, the selection 
unit 357 selects one of the foreground component picture freed of the motion blurring, 
sent from the operating unit 355 and the foreground component picture from the 
motion blurring adding unit 356 adjusted for the motion blurring quantity to output the 
selected foreground component picture. 

The motion blurring adjustment unit 106 thus is able to adjust the motion 
blurring quantity based on the selection signals and the motion blurring adjustment 



"43210 ' 


'foi' 


34321 


F02 


23432 


F03 


12343 


F04 


01234 


F05 



quantity v'. 

For example, if the number of pixels associated with the selection signal is eight 
and the movement quantity v is four, as shown in Fig.61, the motion blurring 
adjustment unit 106 is able to generate the matrix equation (71): 

Sa 

1=03 
05 

•••(71). 

The motion blurring adjustment unit 106 thus establishes anumber of equations 
corresponding to the length of the processing unit to calculate the pixel value Fi 
adjusted for the motion blurring quantity. In similar manner, if the number of pixels 
contained in a processing unit is 100, the motion blurring adjustment unit 106 
generates 100 equations in association with the 100 pixels to calculate Fi. 

In Fig. 62, showing another configuration of the motion blurring adjustment unit 
106, the parts or components similar to those shown in Fig.56 are indicated by the 
same reference numerals and are not explained specifically. 

A selection unit 361 routes the input motion vector and the corresponding 
position signal directly to the processing unit decision unit 351 and to the modelling 
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unit 352. Alternatively, the selection unit 361 substitutes the motion blurring 
adjustment quantity v' for the magnitude of the motion vector to route the motion 
vector, the magnitude of has been replaced by the motion blurring adjustment quantity 
v' and the corresponding position signal directly to the processing unit decision unit 
351 and to the modelling unit 352. 

By so doing, the processing unit decision units 351 to 355 of the motion 
blurring adjustment unit 106 of Fig.62 are able to adjust the motion blurring quantity 
in association with the movement quantity v and with the motion blurring adjustment 
quantity v'. For example, if the movement quantity v is 5 and the motion blurring 
adjustment quantity v' is 3, the processing unit decision units 351 to 355 of the 
motion blurring adjustment unit 106 of Fig.62 executes the processing on the 
foreground component picture, with the movement quantity v of Fig.58 equal to 5, in 
accordance with the model shown in Fig.60 corresponding to the motion blurring 
adjustment quantity v' equal to 3, to calculate a picture containing the motion blurring 
corresponding to the movement quantity v of (movement quantity v)/ (motion blurring 
adjustment quantity v') = 5/3, that is approximately 1.7. Since the calculated picture 
is free of the motion blurring corresponding to the movement quantity v equal to 3, 
attention is to be directed to the fact that the relation between the movement quantity 
V and the motion blurring adjustment quantity v' has a different meaning from the 
results of the motion blurring adding unit 356. 

The motion blurring adjustinent unit 106 generates an equation in association 
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with the movement quantity v and the processing unit and sets the pixel values of the 
foregr9und component picture in the generated equation to calculate the foreground 
component picture adjusted for the motion blurring quantity. 

Referring to the flowchart of Fig.63, the processing for adjusting the motion 
blurring quantity in the foreground component picture by the motion blurring 
adjustinent unit 106 is explained. 

At step S251, the processing unit decision unit 351 of the motion blurring 
adjustment unit 106 generates a processing unit, based on the motion vector and the 
area information, to send the generated processing unit to the modelling unit 352. 

At step S252, the modelling unit 352 of the motion blurring adjustment unit 106 
selects and generates a model in association with the movement quantity v and the 
processing unit. At step S253, the equation generating unit 353 generates the nomial 
equation, based on the selected model. 

At step S254, the addition unit 354 sets pixel values of the foreground 
component picture in the so-generated normal equation. At step S255, the addition 
unit 354 verifies whether or not the pixel values of the totality of pixels of the 
processing imit have been set. If it is verified that the pixel values of the totality of 
pixels corresponding to the processing unit have not been set, the program reverts to 
step S254 to repeat the processing of setting pixel values in the nonnal equation. 

If it is decided at step S255 that the pixel values of the totality of pixels 
corresponding to the processing unit have been set, the program reverts to step S256 
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where the operating unit 355 calculates the pixel values of the foreground, adjusted for 
the motion blurring quantity, based on the nontial equation from the addition unit 354, 
in which the pixel values have been set, to temiinate the processing. 

In this manner, the motion blurring adjustment unit 106 is able to adjust the 
motion blurring quantity based on the motion vector and the area infonnation, from 
the foreground component picture containing the motion blurring. 

That is, the motion blurring adjustment unit 106 is able to adjust the motion 
blurring quantity in the pixel values as sampling data. 

Meanwhile, the structure of the motion blurring adjustment unit 106, shown in 
Fig. 56, is merely exemplary and is not intended to limit the present invention. 

The signal processor 12, the configuration of which is shown in Fig. 10, is able 
to adjust the quantity of the motion blurring contained in the input picture. The signal 
processor 12, the configuration of which is shown in Fig. 10, is able to calculate the 
mixing ratio a as the buried infomiation to output the so-calculated mixing ratio a. 

Fig. 64 is a block diagram showing a modified configuration of the fUnctions of 
the signal processor 12. 

The parts or components similar to those of Fig. 10 are indicated by the same 
reference numerals and are not explained specifically. 

The area specifying unit 103 sends the area information to the mixing ratio 
calculating unit 104 and to the synthesis unit 371. 

The mixing ratio calculating unit 104 sends the area infomiation to the 



m • 

130 

foreground/background separating unit 105 and to the synthesis unit 371. 

The foreground/background separating unit 105 sends the foreground 
component picture to the synthesis unit 371. 

Based on the mixing ratio Oi supplied from the mixing ratio calculating unit 1 04 
and on the area infonnation supplied from the area specifying unit 103, the synthesis 
unit 37 1 synthesizes an optional background picture and the frjreground component 
picture supplied from the fr)reground^ackground separating unit 105 to output a 
picture synthesized from the optional background picture and the ft)reground 
component picture. 

Fig.65 shows the configuration of the sjmthesis unit 371. A background 
component generating unit 381 generates a background component picture, based on 
the mixing ratio a and on an optional background picture, to route the so-generated 
background component picture to a mixed area picture synthesis unit 382. 

The mixed area pictxire synthesis unit 382 synthesizes the background 
component picture supplied from the supplied from the background component 
generating unit 381 and the foreground component picture to generate a mixed area 
synthesized picture which is routed to a picture synthesis unit 383. 

Based on the area information, the picture synthesis unit 383 synthesizes the 
foreground component picture, mixed area synthesized picture supplied from the 
mixed area picture synthesis unit 382 and an optional background picture to generate 
and output a synthesized picture. 
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In this manner, the synthesis unit 371 is able to synthesize the foreground 
component picture to an optional background picture. 

The picture obtained on synthesis of a foreground component picture with an 
optional background picture, based on the mixing ratio cc, as a characteristic value, is 
more spontaneous than a picture obtained on simply synthesizing the pixels. 

Fig.66 shows, in a block diagram, a further configuration of the function of the 
signal processor 12 adapted for adjusting the motion blurring quantity. The signal 
processor 12 shown in Fig. 10 calculates the mixing ratio a and specifies the area 
sequentially, whereas the signal processor 12 shown in Fig.66 specifies the area and 
calculates the mixing ratio a by parallel processing. 

The functions similar to those shown in the block diagram of Fig. 1 0 are denoted 
by the same reference numerals and are not explained specifically. 

The input picture is sent to a mixing ratio calculating unit 401, 
foreground/background separating unit 402, an area specifying unit 103 and to an 
object extraction unit 101. 

Based on the input picture, the mixing ratio calculating unit 40 1 calculates, for 
each of the pixels contained in the input picture, the estimated mixing ratio in case the 
pixel is assumed to belong to the covered background area, and the estimated mixing 
ratio in case the pixel is assumed to belong to the uncovered background area, to 
supply the so-calculated estimated mixing ratio in case the pixel is assumed to belong 
to the covered background area and estimated mixing ratio in case the pixel is assumed 
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to belong to the uncovered background area, to the foreground/background separating 
unit 402. 

Fig.67 shows, in a block diagram an illustrative structure -of the mixing ratio 
calculating unit 401. 

The estimated mixing ratio processor 201, shown in Fig.67, is similar to the 
estimated mixing ratio processor 201 shown in Fig.37. The estimated mixing ratio 
processing unit 202, shown in Fig.67, is the same as the estimated mixing ratio 
processing unit 202 shown in Fig.37. 

The estimated mixing ratio processor 201 calculates the estimated mixing ratio, 
from pixel to pixel, by calculations corresponding to the model of the covered 
background area, based on the input picture, to output the so-calculated estimated 
mixing ratio. 

The estimated mixing ratio processor 202 calculates the estimated mixing ratio, 
from pixel to pixel, by calculations corresponding to the model of the uncovered 
background area, based on the input picture, to output the so-calculated estimated 
mixing ratio. 

Based on the estimated mixing ratio in case the pixel is assumed to belong to 
the covered background area, and the estimated mixing ratio in case the pixel is 
assumed to belong to the uncovered background area, supplied from the mixing ratio 
calculating unit 401, and on the area infonnation, supplied from the area specifying 
unit 103, the foreground/background separating unit 402 generates a foreground 
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component picture from the input picture, to route the so-generated foreground 
component picture to the motion blurring adjustment unit 106 and to the selection unit 
107. 

Fig. 6 8 is a block diagram showing an illustrative structure of the 
foreground/background separating unit 402. 

The parts or components similar to those of the foreground/background 
separating unit 105 shown in Fig.48 are indicated by the same reference numerals and 
not explained specifically. 

Based on the area infonnation supplied from the area specifying unit 103, a 
selection unit 42 1 selects one of the estimated mixing ratio in case the pixel is assumed 
to belong to the covered background area, and the estimated mixing ratio in case the 
pixel is assumed to belong to the uncovered background area, supplied from the 
mixing ratio calculating unit 40 1 , and routes the so-selected estimated mixing ratio as 
the mixing ratio OC to the separating unit 25 1. 

Based on the mixing ratio a and the area infonnation, supplied from the 
selection unit 421, the separating unit 251 separates the foreground components and 
the background components from the pixel values of pixels belonging to the mixed 
area, to send the foreground components extracted to the synthesis unit 253, as well 
as to send the background components to the synthesis unit 255. 

The separating unit 251 may be configured similarly to the structure shown in 

Fig.53. 
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The synthesis unit 253 synthesizes and outputs the foreground component 
picture. The synthesis unit 255 synthesizes and outputs the background component 
picture. 

The motion blurring adjustment unit 106, shown in Fig.66, may be configured 
as in Fig. 1 0 . Based on the area information and the motion vector, the motion blurring 
adjustment unit 106 adjusts the quantity of the motion blurring supplied from the 
foreground/background separating unit 402, to output the foreground component 
picture adjusted for the motion blurring quantity. 

Based on the selection signal, corresponding to the selection by the user, the 
selection unit 107 selects one of the foreground component picture supplied from the 
foreground/background separating unit 402 and the foreground component picture 
from the motion blurring adjustment unit 106, adjusted for the motion blurring 
quantity, to output the selected foreground component picture. 

In this manner, the signal processor 12, the configuration of which is shown in 
Fig. 66, is able to adjust a picture, corresponding to an object of the foreground object 
contained in the input picture, to output the resulting picture. The signal processor 12, 
the configuration of which is shown in Fig.66, is able to calculate the mixing ratio Ct, 
as the buried infonnation, as in the first embodiment, to output the so-calculated 
mixing ratio OC. 

Fig. 6 9 is a block diagram showing a modification of the fiinction of the signal 
processor 12 adapted for synthesizing the foreground component picture to an optional 
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background picture. The signal processor 12, shown in Fig.64, perfonns area 
identification and the calculation of the mixing ratio a in series, whereas the signal 
processor 12, shown in Fig.69, perfonns area identification and the calculation of the 
mixing ratio a in parallel. 

The functions similar to those shown in the block diagram of Fig. 66 are denoted 
by the same reference numerals and are not explained specifically. 

Based on the input picture, the mixing ratio calculating unit 401, shown in 
Fig.69, calculates the estimated mixing ratio for when the pixel is assumed to belong 
to the covered background area and the estimated mixing ratio for when the pixel is 
assumed to belong to the uncovered background area, for each of the pixels contained 
in the input picture, to route the estimated mixing ratio for when the pixel is assumed 
to belong to the covered backgroimd area and the estimated mixing ratio for when the 
pixel is assumed to belong to the uncovered background area, to the 
foreground/background separating unit 402 and to the synthesis unit 43 1. 

Based on the estimated mixing ratio for when the pixel is assumed to belong to 
the covered background area, the estimated mixing ratio for when the pixel is assumed 
to belong to the uncovered background area, supplied from the mixing ratio calculating 
unit 401, and on the area inforaiation supplied from the area specifying unit 103, the 
foreground/background separating unit 402, shown in Fig. 69, generates the foreground 
component picture from the input picture to route the generated foreground component 
picture to the synthesis unit 43 1 . 
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Based on the estimated mixing ratio for when the pixel is assumed to belong to 
the covered background area, the estimated mixing ratio for when the pixel is assumed 
to belong to the uncovered background area, supphed from the mixing ratio calculating 
unit 401, and on the area infonnation supplied from the area specifying unit 103, the 
synthesis unit 431 synthesizes an optional background area and a foreground 
component picture supphed from the foreground/background separating unit 402, to 
output a picture synthesized from the optional background area and the foreground 
component picture. 

Fig. 70 shows the configuration of the synthesis unit 43 1 . The functions similar 
to those shown in the block diagram of Fig.65 are denoted by the same reference 
numerals and are not explained specifically. 

Based on the area infomiation, supplied from the area specifying unit 103, a 
selection unit 441 selects one of the estimated mixing ratio for when the pixel is 
assumed to belong to the covered background area and the estimated mixing ratio for 
when the pixel is assumed to belong to the uncovered background area, supplied from 
the mixing ratio calculating unit 401, to route the selected estimated mixing ratio as 
the mixing ratio a to the background component generating unit 381. 

Based on the mixing ratio CC supplied from the selection unit 441 and the 
optional background component picture, the background component generating unit 
381, shown in Fig.70, generates a background component picture, to route the 
generated picture to the mixed area picture synthesis unit 382. 
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The mixed area picture synthesis unit 382, shown in Fig. 70, synthesizes the 
background component picture, supplied from the background component generating 
unit 381, to the foreground component picture, to generate a mixed area synthesized 
picture, which is routed to the picture synthesis unit 383. 

Based on the area infonnation, the picture synthesis unit 383 synthesizes 
foreground component picture, the mixed area synthesized picture, supplied from the 
mixed area picture synthesis unit 382 and an optional background picture, to generate 
and output a synthesized picture. 

In this manner, the synthesis unit 43 1 is able to synthesize the foreground 
component picture to an optional background picture. 

Although the mixing ratio OC has been explained as a proportion of the 
background component contained in the pixel value, it may also be a proportion of the 
foreground component contained in the pixel value. 

Although the direction of movement of the object as the foreground has been 
explained as being from left to right, this direction is, of course, not limitative. 

An embodiment in which the amount of the motion blurring quantity contained 
in temperature or pressure data by the similar processing as that performed by the 
signal processor 12 is explained. 

Fig. 7 1 shows an illustrative structure of a signal processing apparatus according 
to the present invention. A thermography device 45 1 detects IR rays, radiated from an 
object being measured from an enclosed IR sensor, such as an IR CCD, to generate a 
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signal corresponding to the wavelength or intensity of the detected IR rays. The 
themiography device 45 1 analog/digital converts the generated signal to compare the 
converted signal to reference data corresponding to the reference temperature to 
generate temperature data indicating the temperature of various sites of the object to 
output the generated temperature data to the signal processor 452. 

Similarly to the sensor 11, the thennography device 45 1 has integrating effects 
with respect to the space and time. 

The temperature data the thennography device 451 routes to the signal 
processor 452 is configured similarly to the picture data of the moving picture, and is 
such data in which the values indicating the temperature of respective sites of the 
object being measured (corresponding to the pixel values of the picture data) are 
arrayed two-dimensionally along the spatial direction in association with the picture 
data frames and also are arrayed along the temporal direction. 

The signal processor 452 adjusts the distortion contained in the input 
temperature data and which has been generated as a result of movement of the object 
being measured. For example, the signal processor 452 extracts a more accurate 
temperature of the desired site of the object being measured. 

Fig. 72 is a flowchart showing the processing for adjusting the motion blurring 
quantity by the signal processor 452. At step S30 1, the signal processor 452 acquires 
temperature data in which values indicating the temperatures for respective sites of the 
object being measured are arrayed two-dimensionally. Based on the temperature data. 
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the signal processor 452 generates data specifying the movement. 

At step S302, the signal processor 452 specifies areas of temperature data to a 
foreground area comprising only the values indicating the temperature corresponding 
to a desired object, a background area comprising only the values indicating the 
temperature corresponding to an object other than the desired object, and a mixed area 
comprising the temperature infonnation corresponding to the desired object and the 
temperature information corresponding to the object other than the desired object. 

At step S303, the signal processor 452 checks whether or not the temperature 
indicating value belongs to the temperature data. If the signal processor 452 decides 
that the temperature indicating value belongs to the mixed area, the signal processor 
452 proceeds to step S304 to calculate the mixing ratio 06 by the processing similar to 
that of step SI 02 ofFig.27. 

At step S305, the signal processor 452 separates the information of the 
temperature corresponding to the object desiring temperature measurement, by the 
processing similar to the processing of step S103 of Fig.27, to then proceed to step 
S306. 

For separating the temperature infonnation at step S305, the temperature 
information may be converted, based on the KirchhofP 's law or the law specifying the 
relation between the object temperature and the radiated IR rays, such as Stephen- 
Bohzmann law, into the energy quantity of the IR rays, emitted from the object 
desiring temperature measurement, to separate the energy quantity of the converted 
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IR rays to re-convert the separated energy quantity into temperature. By conversion 
into the IR ray energy prior to separation, the signal processor 452 is able to separate 
the temperature information more accurately than in direct separation of the 
temperature infonnation. 

If, at step S303, the temperature indicating value contained in the temperature 
data does not belong to the mixed area, it is unnecessary to separate the temperature 
infomiation corresponding to the object desiring the temperature measurement. So, 
the processing at steps S304 and S305 are skipped so that the processing proceeds to 
step S306. 

At step S306, the signal processor 452 generates temperature data for causing 
temperature measurement to correspond to the desired object, from a value indicating 
the temperature belonging to the foreground temperature and the information on the 
temperature which causes the temperature measurement to correspond to the desired 
object. 

At step S307, the signal processor 452 generates a model corresponding to the 
generated temperature data by the processing similar to the processing at step S25 1. 

At step S308, the signal processor 452 adjusts the quantity of the motion 
blurring contained in the temperature data corresponding to the object, in need of 
temperature measurement, by the processing similar to that of steps S252 to S255 of 
Fig.63, based on the generated model, to teraiinate the processing. 

In this manner, the signal processor 452 adjusts the quantity of the motion 
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blurring contained in the temperature data generated by the movement of the object 
being measured to calculate the more accurate temperature difference of respective 
object portions. 

Fig.73 shows an illustrative structure of a signal processing apparatus according 
to the present invention for weighing measurement. The pressure area sensor 501 is 
made up of plural pressure sensors to measure the load per unit planar area, that is the 
pressure. . The pressure area sensor 501 is of a structure comprised of a two- 
dimensional array on a floor surface of plural pressure sensors 51 1-1-1 to 51 1-M-N. 
When an object 5 12, the weight of which is being measured, is moved on the pressure 
area sensor 501, the pressure area sensor 501 measures the pressure applied to each 
of the pressure sensors 511-1-1 to 511-M-N to generate weight data for each of 
measured ranges of the pressure sensors 5 11- 1 -1 to 5 1 1-M-N to output the generated 
weight data to the signal processor 502. 

The pressure sensors 511-1-1 to 511-M-N are each made up of a sensor 
exploiting double refraction produced when an external force is applied to a 
transparent elastic material, or the so-called photoelasticity. 

The pressure area sensor 501 in its entirety may be constructed by a sensor 
exploiting the photoelasticity. 

Fig.75 illustrates the load associated with the weight of each part of the object 
512 apphed to respective ones of the pressure sensors 511-1-1 to 511 -M-N making up 
the pressure area sensor 501. 
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The load A corresponding to the weight of the leftmost portion of the object 5 1 2 
in Fig.75 is applied to the pressure sensor 51 1-m-l. The load b corresponding to the 
weight of the second left portion of the object 512 is apphed to the pressure sensor 
51 l-m-2. The load c corresponding to the weight of the fourth left portion of the 
object 5 12 is applied to the pressure sensor 5 1 l-m-3. The load d corresponding to the 
weight of the fourth left portion of the object 5 12 is apphed to the pressure sensor 511- 
m-4. 

The load e corresponding to the weight of the second left portion of the object 
5 12 is applied to the pressure sensor 5 1 l-m-5. The load f corresponding to the weight 
of the fourth left portion of the object 512 is apphed to the pressure sensor 5 1 l-m-6. 
The load g corresponding to the weight of the fourth left portion of the object 5 12 is 
applied to the pressure sensor 5 ll-m-7. 

The weight data output by the pressure area sensor 501 corresponds to the 
arrangement of the pressure sensor 51 1-1-1 to 51 1-M-N and is comprised of weight 
values arrayed two-dimensionally in the spatial direction. 

Fig. 76 illustrates typical weight data output by the pressure area sensor 501 
when the object 512 is moving, with the pressure area sensor 501 having integrating 
effects. 

On the pressure sensor 5 1 1-m- 1 is appHed a load A corresponding to the weight 
of the leftmost portion of the object 512 in a unit tune of measurement, a value A is 
output as a value indicating the weight included in the weight data. 
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In the unit time for measurement, there are applied to the pressure sensor 511- 
m-2 a load b corresponding to the weight of the second left portion of the object 512, 
and a load d corresponding to the weight of the leftmost portion of the object 5 12,so 
the pressure sensor 511-m-2 outputs a value A+b as a value indicating the weight 
comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-3 a load c corresponding to the weight of the third left portion of the object 5 12, a 
load b corresponding to the weight of the second left portion of the object 512 and 
subsequently the load A corresponding to the weight of the second left portion of the 
object 512, so the pressure sensor 511-m-2 outputs a value A+b+c as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-4 a load d corresponding to the weight of the fourth left portion of the object 5 12, 
a load c corresponding to the weight of the third left portion of the object 5 12, a load 
b corresponding to the weight of the second left portion of the object 512 and 
subsequently the load A corresponding to the weight of the leftmost portion of the 
object 512, so the pressure sensor 511-m-2 outputs a value A+b+c+d as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-5 a load e corresponding to the weight of the fifth left portion of the object 5 12, a 
load d corresponding to the weight of the fourth left portion of the object 5 12, a load 
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c corresponding to the weight of the third left portion of the object 512 and 
subsequently the load A corresponding to the weight of the second left portion of the 
object 512, so the pressure sensor 511-m-5 outputs a value b+c+d+e as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are apphed to the pressure sensor 511- 
m-6 a load f corresponding to the weight of the sixth left portion of the object 5 12, a 
load e corresponding to the weight of the fifth left portion of the object 5 12, a load d 
corresponding to the weight of the fourth left portion of the object 512 and 
subsequently the load c corresponding to the weight of the third left portion of the 
object 512, so the pressure sensor 511-m-6 outputs a value c+d+e+f as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 5 1 1- 
m-7 a load g corresponding to the weight of the seventh left portion of the object 512, 
a load f corresponding to the weight of the sixth left portion of the object 5 12, a load 
e corresponding to the weight of the fifth left portion of the object 512 and 
subsequently the load d corresponding to the weight of the fourth left portion of the 
object 512, so the pressure sensor 511-m-7 outputs a value d+e+f+g as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-8 a load g corresponding to the weight of the seventh left portion of the object 5 12, 
a load f corresponding to the weight of the sixth left portion of the object 512, and a 
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load e corresponding to the weight of the fifth left portion of the object 512, so the 
pressure sensor 51I-m-8 outputs a value e+f+g as a value indicating the weight 
comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-9 a load g corresponding to the weight of the seventh left portion of the object 5 12, 
and a load f corresponding to the weight of the sixth left portion of the object 5 12, so 
the pressure sensor 511-m-9 outputs a value f+g as a value indicating the weight 
comprehended in the weight data. 

In the unit time for measurement, there is applied to the pressure sensor 511 -m- 
10 a load g corresponding to the weight of the seventh left portion of the object 5 12, 
so the pressure sensor 511-m-lO outputs a value g as a value indicating the weight 
comprehended in the weight data. 

The pressure area sensor 50 1 outputs weight data comprised of the value A 
output by the pressure sensor 51 1-m-l, the value A+b output by the pressure sensor 
511 -m-2, the value A+b+c output by the pressure sensor 5 1 l-m-3, the value A+b+c+d 
output by the pressure sensor 511-m-4, the value b+c+d+e output by the pressure 
sensor 51 l-m-4, the value A+b+c output by the pressure sensor 51 l-m-3, the value 
A+b+c+d output by the pressure sensor 5 1 l-m-4, the value b+c+d+e output by the 
pressure sensor 5 1 l-m-5, the value c+d+e+f output by the pressure sensor 5 1 l-m-6, 
the value d+e+f+g output by the pressure sensor 511 -m-7, the value e+f+g output by 
the pressure sensor 5 1 l-m-8, the value f+g output by the pressure sensor 5 1 l-m-9. 
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and the value g output by the pressure sensor 51 1-m-lO. 

The signal processor 502 adjusts the distortion generated by the movement of 
the object 512 being measured from the weight data supphed from the pressure area 
sensor 501. For example, the signal processor 502 extracts more accurate weight of 
the desired sites of the object 512 being measured. For example, the signal processor 
502 extracts the loads A and b to g from weight data comprised of the value A, A+b, 
A+b+c, A+b+c+d, b+c+d+e, c+d+e+f, d+e+f+g, e+f+g, f+g and g. 

Referring to the flowchart of Fig. 77, the processing for calculating the load 
executed by the signal processor 502 is explained. 

At step S401, the signal processor 502 acquires weight data output by the 
pressure area sensor 50 1 . At step S402, the signal processor 502 decides, based on the 
weight data acquired from the pressure area sensor 501, whether or not the load of the 
object 512 is being applied to the pressure area sensor 501. If it is decided that the 
load of the object 512 is being applied to the pressure area sensor 501, the signal 
processor 502 proceeds to step S403 to acquire the movement of the object 512 based 
on changes in the weight data. 

At step S404, the signal processor 502 acquires one-line data of the pressure 
sensor 511 contained in the weight data along the direction of movement acquired by 
the processing at step S403. 

At step S405, the signal processor 502 calculates the load corresponding to the 
weight of the respective portions o the object 512 to tenninate the processing. The 
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signal processor 502 calculates the load corresponding to the weights of respective 
parts of the object 512 by a processing similar to the processing explained with 
reference to the flowchart of Fig.63. 

If, at step S402, the load of the object 5 12 is not appHed to the pressure area 
sensor 501, there is no weight data to be processed, so the processing is tenninated. 

In this manner, the weight measurement system is able to calculate the correct 
load corresponding to the weight of each portion of the moving object. 

The signal processor 12 generating a picture of higher resolution in the spatial 
direction is explained. 

Fig. 7 8 is a block diagram showing a configuration of generating high resolution 
picture by increasing the number of pixels per frame, as another function of the signal 
processor 12. 

A frame memory 701 stores an input picture on the frame basis and sends the 
stored picture to a pixel value generator 702 and to a correlation calculating unit 703. 

The correlation calculating unit 703 calculates the correlation values of pixel 
values neighboring to one another in a transverse direction, contained in a picture 
supplied from the frame memory 701, to send the calculated correlation values to the 
pixel value generator 702. The pixel value generator 702 calculates double density 
picture components from the pixel values of the center pixel, based on the correlation 
values supplied from the correlation calculating unit 703, to generate a horizontal 
double-density picture, with the so-calculated picture component as the pixel value. 
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The pixel value generator 702 sends the so-generated horizontal double-density picture 
to the frame memory 704. 

The frame memory 704 stores the horizontal double-density picture, supplied 
from the pixel value generator 702, to send the so-stored horizontal double-density 
picture to a pixel value generating unit 705 and to a correlation calculating unit 706. 

The correlation calculating unit 706 calculates the correlation values of pixel 
values neighboring to one another in a vertical direction, contained in a picture 
supplied from the frame memory 704, to send the calculated correlation values to the 
pixel value generator 705. The pixel value generator 705 calculates double density 
picture components from the pixel values of the center pixel, based on the correlation 
values supplied from the correlation calculating unit 703, to generate a horizontal 
double-density picture, with the so-calculated picture component as the pixel value. 
The pixel value generator 705 outputs the so-generated horizontal double-density 
picture. 

The processing for generating a horizontal double-density picture by the pixel 
value generator 702 is explained. 

Fig. 79 shows an arrangement of pixels provided in a sensor 1 1 as a CCD, and 
an area for pixel data of a horizontal double-density picture. In Fig.79, A to I indicate 
individual pixels. The areas A to r each denote a light reception area obtained on 
halving the individual pixels A to I in the longitudinal direction. With the width 21 of 
the light reception area of each of the pixels A to I, the width of each of the areas A 
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to r is I. The pixel value generator 702 calculates the pixel values of the pixel data 
associated with the areas A to r. 

Fig. 80 illustrates pixel data corresponding to the light incident on the areas A 
to r. In Fig. 80, f (x) denotes an spatially ideal pixel value in association with the input 
light and to the spatially tiny domain. 

If a pixel value of a pixel data is represented by the unifonn integration of the 
ideal pixel value f (x), the pixel value Yl of pixel data associated with the area i is 
represented by the equation (72): 

Yl-f{x)d.-\ ...(72) 
whilst the pixel value Y2 of the picture data associated with the area j and the pixel 
value Y3 of the pixel E are represented by the following equations (73) and (74): 



Y2 = \llf[x}dx-\ 



- (73) 

and 



2e 
71+ 72 

-(74) 

respectively. 

In the above equations (72) to (74), xl, x2 and x3 are spatial coordinates of the 
respective boundaries of the light reception area, area i and the area j of the pixel E, 
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respectively. 

By modifying the equation (74), the following equations (75), (76) may be 
derived: 

Yl = 2-Y3-Y2 

•••(75) 

and 

Y2 = 2-Y3-Y1 

•••(76). 

Therefore, if the pixel value Y3 of the pixel E and the pixel value Y2 of the 
pixel data corresponding to the area j are known, the pixel value Yl of the pixel data 
corresponding to the area i may be calculated from the equation (75). Also, if the pixel 
value Y3 of the pixel E and the pixel value Yl of the pixel data corresponding to the 
area i are known, the pixel value Y2 of the pixel data corresponding to the area j can 
be calculated from the area j. 

If the pixel value corresponding to a pixel and one of pixel values of the pixel 
data corresponding to the two areas of the pixel are known, the pixel value of the other 
pixel data corresponding to the other areas of the pixel may be calculated. 

Referring to Fig. 8 1 , the manner of calculating the pixel values of the pixel data 
corresponding to the two areas of one pixel is explained. Fig.81 A shows the relation 
between the pixels D, E and F and the spatially ideal pixel value f (x). 

Since the pixels D to F own integrating effects, and one pixel outputs a pixel 
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value, each one pixel value is output, as shown in Fig. 8 IB. The pixel value output by 
the pixel E corresponds to the integrated value of the pixel value f (x) in the range of 
the light reception area. 

The correlation calculating unit 703 generates the correlation value between the ' 
pixel value of the pixel D and that of the pixel E, and the correlation value between the 
pixel value of the pixel E and that of the pixel F, to route the so-generated correlation 
value to the pixel value generator 702. The correlation value calculated by the 
correlation calculating unit 703 is calculated based on the difference between the pixel 
value of the pixel D and that of the pixel E, or on the difference between the pixel 
value of the pixel E and that of the pixel F. When the pixel values of neighboring 
pixels are closer to each other, these pixels may be said to have higher correlation. 
That IS, a smaller value of the difference between pixel values indicate stronger 
correlation. 

So, if the difference between the pixel value of the pixel D and that of the pixel 
E, or the difference between the pixel value of the pixel E and that of the pixel F, is 
directly used as a correlation value, the correlation value, which is the smaller 
difference value, exhibits stronger correlation. 

For example, if the correlation between the pixel value of the pixel D and that 
of the pixel E are stronger than that between the pixel value of the pixel E and that of 
the pixel F, the pixel value generator 702 divides the pixel value of the pixel D by 2 
to use the resulting value as pixel data of the area i. 
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Based on the pixel value of the pixel E and on the pixel value of the area i, the 
pixel value generator 702 calculates the pixel values of the pixel data of the area j in 
accordance with the equation (75) or (76), as shown in Fig.SlD. 

The pixel value generator 702 calculates the pixel values of the pixel data of the 
area g and those of the pixel data of the area h, for e.g., the pixel D, to calculate the 
pixel value of the pixel data of the area i and the pixel value of the pixel data of the 
area j, and then to calculate the pixel value of the pixel data of the area k and the pixel 
value of the pixel data of the area 1, and so on, to calculate the pixel values of the pixel 
data in the picture as described above to generate a horizontal double-density picture 
comprehending the pixel values of the pixel data calculated to furnish the so-generated 
horizontal double-density picture to the frame memory 704. 

Similarly to the pixel value generator 702, the pixel value generating unit 705 
calculates, from the correlation of the pixel values of three vertically arrayed pixels of 
the horizontal double-density picture, supplied from the correlation calculating unit 
706, and from the pixel values of the three pixels, the pixel values of the picture data 
corresponding to two areas obtained on vertically splitting the light reception area of 
the pixel, to thereby generate the double density picture. 

When fed with the picture shown as an example in Fig. 82, the pixel value 
generating xmit 702 generates a double-density picture shown as an example in Fig. 83. 

When fed with a picture, shown as an example in Fig. 82, the pixel value 
generating unit 705 generates a picture, shown as an example in Fig.84. When fed 
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with a horizontal double-density picture, shown as an example in Fig.83, the pixel 
value generating unit 705 generates a double-density picture, shown as an example in 
Fig.85. 

Fig. 86 is a flowchart for illustrating the processing for generating the double- 
density picture by the signal processor 12, a structure of which is shown in Fig.78. At 
step S601, the signal processor 12 acquires an input picture to store it in the frame 
memory 701. 

At step S602, the correlation calculating unit 703 selects one of the pixels in the 
picture as a considered pixel, and finds a pixel horizontally neighboring to the 
considered pixel, based on the pixel value stored in the frame memory 701. At step 
S603, the pixel value generator 702 generates a pixel value of pixel data lying on one 
side of the horizontal double-density picture from a pixel value exhibiting stronger 
correlation, that is a higher correlation value. 

Based on the characteristics of the CCD, the pixel value generator 702 at step 
S604 generates pixel values of other pixel data of the horizontal double-density 
picture. Specifically, the pixel value generator 702 calculates pixel values of the other 
picture data of the horizontal double-density picture, based on the pixel value 
calculated by the processing of step S603 and on the pixel value of the picture data of 
the input picture, in accordance with the equations (75) and (76) explained with 
reference to Fig. 80. The picture data of the horizontal double-density picture for the 
considered pixel, generated by the processing at steps S603 and S604, are stored in the 
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frame memory 704. 

At step S605, the pixel value generator 702 checks whether or not the 
processing of the entire picture has come to a close. If it is determined that the 
processing of the entire picture has come to a close, the program reverts to step S602 
to select the next pixel as the considered pixel to repeat the processing of generating 
the horizontal double-density picture. 

If it is detennined at step S605 that the processing of the entire picture has come 
to a close, the correlation calculating unit 706 selects one of the pixels in the picture 
as the considered pixel to find the correlation value of the pixel neighboring to the 
considered pixel in the vertical direction based on the pixel value of the horizontal 
double-density picture stored in the frame memory 704. At step S607, the pixel value 
generating unit 705 generates the pixel value of the one side of the double-density 
picture from the pixel values of the stronger correlation, based on the correlation value 
supplied from the correlation calculating unit 706. 

At step S608, as at step S604, the pixel value generating unit 705 generates the 
other pixel value of the double-density picture, based on the characteristics of the 
CCD. Specifically, the pixel value generator 702 calculates the pixel values of the 
other picture data of the double-density picture, based on the pixel values calculated 
by the processing at step S607 and on the pixel value of the pixel data of the horizontal 
double-density picture, in accordance with the equations (75) and (76) explained with 
reference to Fig. 80. 
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At step S609, the pixel value generating unit 705 decides whether or not the 
processing of the entire picture has been finished. If it is decided that the processing 
of the entire picture has not been finished, the program reverts to step S606 to select 
the next pixel as the considered pixel to repeat the processing of generating the 
double-density picture. 

If it is decided at step S609 that the processing of the entire picture has been 
finished, the pixel value generating unit 705 outputs the so-generated double-density 
picture to complete the processing. 

In this manner, a double-density picture, the number of pixels of which in the 
vertical and in the horizontal directions are doubled, may be produced fi-om the input 
picture by the signal processor 12, the structure of which is shown in Fig.78. 

The signal processor 12, the structure of which is shown in Fig.78, is able to 
generate a picture of high spatial resolution by perfonning signal processing taking 
account of the pixel correlation and the integrating effect of the CCD with respect to 
the space. 

In the foregoing, a picture of the real space having a three-dimensional space 
and the time axis infonnation is mapped on a time space having the two-dimensional 
space and the time axis information using a video camera. The present invention is, 
however, not limited to this embodiment and may be apphed to correction of distortion 
caused by projecting the first information of a higher order first dimension to the 
lower-order second dimension, extraction of the significant information or to synthesis 
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of more spontaneous pictures. 

The sensor 1 1 may also be a sensor exemplified by, for example, a BBD (bucket 
brigard device), a CID (charge injection device) or a CDD (charge priming device), 
without being limited to a CCD. The sensor 1 1 may also be a sensor in which 
detection devices are arranged in a row instead of in a matrix. 

The recording medium, having recorded thereon a program for executing the 
signal processing of the present invention may not only be constructed by a package 
medium, distributed to users for fiimishing the program separately from a computer, 
inclusive of a magnetic disc 5 1 , such as a floppy disc, having the program pre-recorded 
thereon, an optical disc 52, such as CD-ROM, Compact Disc, read-only memory or 
DVD (digital versatile disc), a magneto-optical disc 53, such as MD (mini-disc) or a 
semiconductor memory 54. But may also be constructed by a ROM 22 furnished to the 
user in a pre-assembled state in a computer, and having the program recorded thereon, 
and a hard disc included in the memory unit 28. 

It should be noted that, in the present specification, the step for stating a 
program recorded on a recording medium includes not only the processing carried out 
chronologically in the specified sequence but also the processingthat is not necessarily 
processed chronologically but is executed in parallel or batch-wise. 

Thus, based on the area infonnation specifying a foreground area made up only 
of foreground object components making up a foreground object in the picture data, 
a background area made up only of background object components making up a 
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background object in the picture data, and on a mixed area which is a mixture of the 
foreground object components and the background object components in the picture 
data, and on the picture data, the mixed area including a covered background area 
formed at a leading end in the movement direction of the foreground object, and an 
uncovered background area fonned at a trailing end of the foreground object, a 
processing unit made up of pixel data lying on at least a straight line extending in a 
direction coincident with the direction of movement of the foreground object from an 
outer end of the covered background area to an outer end of the uncovered background 
area, centered about the foreground area, are set. Anonnal equation is then generated 
by setting pixel values of pixels in the processing unit decided based on the processing 
unit and an unknown dividing value obtained on dividing the foreground object 
components in the mixed area with a predetennined dividing number. This normal 
equation is solved by the least square method to generate foreground object 
components adjusted for the quantity of movement blurring to adjust the movement 
blurring quantity. 

Also, sample data present in detection data lying ahead and at back of the 
considered detection data where there exist considered sample data which is the 
sample data under consideration is extracted as foreground sample data corresponding 
to the foreground in the real world, whilst sample data present in detection data lying 
ahead and at back of the considered detection data where there exist considered 
sample data which is the sample data under consideration is extracted as background 
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sample data corresponding to the background in the real world, and the mixing ratio 
of the considered sample data is detected based on the considered sample data, 
foreground sample data and on the background sample data, thereby enabling the 
detection of the mixing ratio. 

The still/movement decision is given, based on the detection data, and a mixed 
area containing sample data comprised of a mixture of plural objects in the real world 
is detected, thereby enabling detection of the mixing ratio. 

The second signal of a second dimension is acquired by detecting the first signal 
of the real world having a first dimension as mapped on the sensor, with the second 
dimension being lower than the first dimension, and the signal processing is performed 
on the second signal, thereby enabling the significant infonnation buried due to 
projection to be extracted from the second signal. 

Since the second signal is acquired by detecting the first signal of the real world 
having the first dimension by the sensor, with the second signal being of a second 
dimension lower than the first dimension and presenting distortion relative to the first 
signal, and a third signal, alleviated in distortion relative to the second signal, is 
generated by processing based on the second signal, it is possible to alleviate the signal 
distortion. 

In the detection signal, the foreground area, composed only of the foreground 
object components, constituting the foreground object, the background area, composed 
only of the background object components, constituting the background object, and 
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the mixed area composed of the foreground object components and the background 
object components, are specified, the mixing ratio of the foreground object 
components and background object components at least in the mixed area is detected, 
and the foreground object components and background object components are 
separated from each other based on the specified results and on the mixing ratio, thus 
enabling utilization of the foreground and background objects as data of higher quality. 

In the detection signal, the foreground area, composed only of the foreground 
object components, constituting the foreground object, the background area, composed 
only of the background object components, constituting the background object, and 
the mixed area composed of the foreground object components and the background 
object components, are specified, and the mixing ratio of the foreground and 
background object components in at least the mixed area is determined based on the 
specified results, thus enabling detection of the mixing ratio as the significant 
infonnation. 

The mixing ratio of the foreground and background object components in the 
mixed area comprised of a mixture of the foreground object components, constituting 
the foreground object, and the background object components, constituting the 
background object, is detected, and the foreground and background object components 
are separated from each other based on the mixing ratio, thus enabling utilization of 
the foreground and background objects as data of higher quality. 



